Stephen J. Turnbull wrote:

Krinetzki, Stephan writes:

...

Hi everyone, any news here? We are running with Pyton 3.6 - maybe a Python Upgrade can speed Mailman up? It would help if you provide the original description. I see you're running Apache + uswgi. uwsgi has been pretty finicky, and most of the developers use gunicorn as the WSGI host. I suggest you ask on the uswgi channels as well. (I don't think that's a *better* place to ask, just a relevant place.)

Then we should maybe move to gunicorn, if this is the better way (or at least better supported way)

You wrote:

...

If we try to approve them via Postorius, the message is delivered to the list immediately, but the success message in Postorius takes a long time - on average the process hangs for three minutes until the message is no longer displayed. You jump to the conclusion that the REST API is slow, but there's zero evidence to localize to the REST API. If your browser isn't running on the same host as the webserver it could even be some kind of network timeout (reverse DNS?) completely unrelated to Mailman (unlikely, but can't be ruled out with the information provided).

What is in the various logs? Mailman core, Django, your backend database, the webserver, the WSGI hosts all may be writing to logs, especially if there's some kind of time out or other error condition.

After we enabled the "slow query log", we got in the postgresql database the following output:

2022-01-25 12:49:41.538 CET [16126] LOG:  unexpected EOF on client connection with an open transaction
2022-01-25 15:34:36.047 CET [16864] LOG:  duration: 30760.028 ms  statement: DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX

2022-01-25 15:34:36.047 CET [3900] LOG: duration: 9372.471 ms statement: DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX 2022-01-25 16:42:07.243 CET [3900] LOG: unexpected EOF on client connection with an open transaction

Pretty sounds like a database problem?

The other logs are silent after we set the following values in the gunicorn.cfg in the mailman home:

[gunicorn]
# All the options that can be put here are available at
# http://docs.gunicorn.org/en/stable/settings.html#settings
# No of seconds to wait before killing the worker processes.
graceful_timeout = 30
# Timeout for the requests.
timeout = 360

sometimes the log (mailmansuite.log) show us this error

ERROR 2022-01-25 18:55:55,037 2599 django.request Service Unavailable: /postorius/lists/games.lists.rwth-aachen.de/held_messages
ERROR 2022-01-25 18:58:24,325 2588 postorius Mailman REST API not available
Traceback (most recent call last):
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/util/retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/lib64/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restbase/connection.py", line 145, in call
    response = request(**params, auth=self.auth)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/postorius/auth/decorators.py", line 63, in wrapper
    return fn(*args, **kwargs)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/postorius/views/list.py", line 698, in list_moderation
    _perform_action(message_ids, mailing_list.accept_message)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/postorius/views/list.py", line 685, in _perform_action
    action(message_id)
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restobjects/mailinglist.py", line 364, in accept_message
    return self.moderate_message(request_id, 'accept')
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restobjects/mailinglist.py", line 334, in moderate_message
    path, data, 'POST')
  File "/opt/mailman/mailman-venv/lib64/python3.6/site-packages/mailmanclient/restbase/connection.py", line 169, in call
    'Could not connect to Mailman API: ', repr(e))
mailmanclient.restbase.connection.MailmanConnectionError: ('Could not connect to Mailman API: ', "ConnectionError(ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)),)")

* The venv is in /opt/mailman/mailman-venv

• The settings are in /opt/mailman/mailman-venv/mailman-suite/mailman-suite_project

content of mailman.cfg:

# AUTOMATICALLY GENERATED BY MAILMAN ON 2016-09-12 14:35:56
#
# This is your GNU Mailman 3 configuration file.  You can edit this file to
# configure Mailman to your needs, and Mailman will never overwrite it.
# Additional configuration information is (for now) available in the
# schema.cfg file <http://tinyurl.com/cm5rtqe> and the base mailman.cfg file
# <http://tinyurl.com/dx9b8eg>.
#
# For example, uncomment the following lines to run Mailman in developer mode.
#
# [devmode]
# enabled: yes
# recipient: your.address@your.domain

[mailman]
site_owner: mailman@lists.example.com
layout: here
pending_request_life: 3d
#layout: fhs
#[general]
#hostname = 'lists.example.com'

# The default language for this server.
default_language: de

[paths.here]
var_dir: /opt/mailman/var
log_dir: /var/log/mailman/mailman-logs

[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://mailman:<PASS>@127.0.0.1/mailman
debug: no

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /opt/mailman/mailman-suite/mailman-suite_project/mailman-hyperkitty.cfg

#[archiver.mail_archive]
#enable: yes

#[plugin.example]
#class: example.hooks.ExamplePlugin
#enabled: yes

#[archiver.prototype]
#enable: yes

[webservice]
# The hostname at which admin web service resources are exposed.
hostname: localhost
# The port at which the admin web service resources are exposed.
port: 8000
# Whether or not requests to the web service are secured through SSL.
use_https: no
# Whether or not to show tracebacks in an HTTP response for a request that
# raised an exception.
show_tracebacks: yes
# The API version number for the current (highest) API.
api_version: 3.1
# The administrative username.
admin_user: mailmanapi
# The administrative password.
admin_pass: <PASS>
workers: 4
configuration: /opt/mailman/gunicorn.cfg

#[language.master]
[language.de]
description: German
#charset: iso-8859-1
charset: utf-8
enabled: yes

[language.en]
description: English
charset: utf-8
enabled: yes

[mta]
incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: 127.0.0.1
smtp_port: 25
configuration: /opt/mailman/mailman-venv/lib/python3.6/site-packages/mailman/config/postfix.cfg
verp_probes: yes

[logging.root]
level: debug
path: mailman.log
[logging.archiver]
level: warn
path: archiver.log
[logging.bounce]
path: bounce.log
level: warn
[logging.config]
level: debug
path: mailman.log
[logging.database]
level: warn
path: database.log
[logging.debug]
path: debug.log
level: debug
[logging.error]
level: debug
path: error.log
[logging.fromusenet]
level: warn
path: mailman.log
[logging.http]
level: warn
path: httpd.log
[logging.locks]
level: warn
path: locks.log
[logging.mischief]
level: warn
path: mailman.log
[logging.plugins]
path: mailman.log
level: warn
[logging.runner]
level: debug
path: runner.log
[logging.smtp]
path: smtp.log
level: debug
[logging.subscribe]
path: subscribe.log
level: warn
[logging.vette]
path: vette.log
level: warn

# Some list posts and mail to the -owner address may contain DomainKey or
# DomainKeys Identified Mail (DKIM) signature headers <http://www.dkim.org/>.
# Various list transformations to the message such as adding a list header or
# footer or scrubbing attachments or even reply-to munging can break these
# signatures.  It is generally felt that these signatures have value, even if
# broken and even if the outgoing message is resigned.  However, some sites
# may wish to remove these headers by setting this to 'yes'.
remove_dkim_headers: yes

[bounces]
# How often should the bounce runner process queued detected bounces?
register_bounces_every: 15m

contenten of settings.py

# -*- coding: utf-8 -*-
# Copyright (C) 1998-2016 by the Free Software Foundation, Inc.
#
# This file is part of Mailman Suite.
#
# Mailman Suite is free sofware: you can redistribute it and/or modify it
# under the terms of the GNU General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option)
# any later version.
#
# Mailman Suite is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
# or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
# for more details.

# You should have received a copy of the GNU General Public License along
# with Mailman Suite.  If not, see <http://www.gnu.org/licenses/>.
"""
Django Settings for Mailman Suite (hyperkitty + postorius)

For more information on this file, see
https://docs.djangoproject.com/en/1.8/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/1.8/ref/settings/
"""

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
import os

BASE_DIR = os.path.dirname(os.path.abspath(__file__))

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'hdeu37xh3291kf'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False

ADMINS = (
     ('Mailman Admin', 'mailman@lists.example.com'))


SITE_ID = 1

# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.8/ref/settings/#allowed-hosts
ALLOWED_HOSTS = [
    "lists.example.com", "example.com", "localhost", "127.0.0.1", "XXX.XXX.XXX.XXX", "YYY.YYY.YYY.YYY" # Archiving API from Mailman, keep it.
    # "lists.your-domain.org",
    # Add here all production URLs you may have.
]

# Mailman API credentials
MAILMAN_REST_API_URL = 'http://localhost:8000'
MAILMAN_REST_API_USER = 'mailmanapi'
MAILMAN_REST_API_PASS = '<PASS>'
MAILMAN_ARCHIVER_KEY = '<ARCHIVER_KEY>'
MAILMAN_ARCHIVER_FROM = ('XXX.XXX.XXX.XXX', 'localhost', '127.0.0.1', '::1')

# Application definition

INSTALLED_APPS = (
    'hyperkitty',
    'postorius',
    'django_mailman3',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'rest_framework',
    'django_gravatar',
    'compressor',
    'haystack',
    'django_extensions',
    'django_q',
    'allauth',
    'allauth.account',
    'allauth.socialaccount',
    #'django_mailman3.lib.auth.fedora',
    #'allauth.socialaccount.providers.openid',
    #'allauth.socialaccount.providers.github',
    #'allauth.socialaccount.providers.gitlab',
    #'allauth.socialaccount.providers.google',
    ## 'allauth.socialaccount.providers.facebook',
    #'allauth.socialaccount.providers.twitter',
    #'allauth.socialaccount.providers.stackexchange',
)


MIDDLEWARE = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django_mailman3.middleware.TimezoneMiddleware',
    'postorius.middleware.PostoriusMiddleware',
)

ROOT_URLCONF = 'urls'


TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.i18n',
                'django.template.context_processors.media',
                'django.template.context_processors.static',
                'django.template.context_processors.tz',
                'django.template.context_processors.csrf',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'django_mailman3.context_processors.common',
                'hyperkitty.context_processors.common',
                'postorius.context_processors.postorius',
            ],
        },
    },
]

WSGI_APPLICATION = 'wsgi.application'


# Database
# https://docs.djangoproject.com/en/1.8/ref/settings/#databases

DATABASES = {
    #'default': {
        # Use 'sqlite3', 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        #'ENGINE': 'django.db.backends.sqlite3',
        # DB name or path to database file if using sqlite3.
        #'NAME': os.path.join(BASE_DIR, 'mailmansuite.db'),
        ## The following settings are not used with sqlite3:
        #'USER': 'mailmansuite',
        #'PASSWORD': 'mmpass',
        # HOST: empty for localhost through domain sockets or '127.0.0.1' for
        # localhost through TCP.
        #'HOST': '',
        # PORT: set to empty string for default.
        #'PORT': '',
    #}
    # Example for PostgreSQL (recommanded for production):
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'mailman',
        'USER': 'mailman',
        'PASSWORD': '<PASS>',
        'HOST': 'localhost',
    }
}

# Maintain type of autogenerated keys going forward
# https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'


# If you're behind a proxy, use the X-Forwarded-Host header
# See https://docs.djangoproject.com/en/1.8/ref/settings/#use-x-forwarded-host
# USE_X_FORWARDED_HOST = True

# And if your proxy does your SSL encoding for you, set SECURE_PROXY_SSL_HEADER
# https://docs.djangoproject.com/en/1.8/ref/settings/#secure-proxy-ssl-header
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_SCHEME', 'https')

# Other security settings
# SECURE_SSL_REDIRECT = True
# If you set SECURE_SSL_REDIRECT to True, make sure the SECURE_REDIRECT_EXEMPT
# contains at least this line:
# SECURE_REDIRECT_EXEMPT = [
#     "archives/api/mailman/.*",  # Request from Mailman.
#     ]
# SESSION_COOKIE_SECURE = True
# SECURE_CONTENT_TYPE_NOSNIFF = True
# SECURE_BROWSER_XSS_FILTER = True
# CSRF_COOKIE_SECURE = True
# CSRF_COOKIE_HTTPONLY = True
# X_FRAME_OPTIONS = 'DENY'


# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME':
'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME':
'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME':
'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME':
'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Internationalization
# https://docs.djangoproject.com/en/1.8/topics/i18n/

#LANGUAGE_CODE = 'en-us'
LANGUAGE_CODE = 'de-de'

#TIME_ZONE = 'UTC'
TIME_ZONE = 'Europe/Berlin'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.8/howto/static-files/

# Absolute path to the directory static files should be collected to.
# Don't put anything in this directory yourself; store your static files
# in apps' "static/" subdirectories and in STATICFILES_DIRS.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = os.path.join(BASE_DIR, 'static')

# URL prefix for static files.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'

# Additional locations of static files
STATICFILES_DIRS = (
    # Put strings here, like "/home/html/static" or "C:/www/django/static".
    # Always use forward slashes, even on Windows.
    # Don't forget to use absolute paths, not relative paths.
    # BASE_DIR + '/static/',
)

# List of finder classes that know how to find static files in
# various locations.
STATICFILES_FINDERS = (
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    # 'django.contrib.staticfiles.finders.DefaultStorageFinder',
    'compressor.finders.CompressorFinder',
)

# Django 1.6+ defaults to a JSON serializer, but it won't work with
# django-openid, see
# https://bugs.launchpad.net/django-openid-auth/+bug/1252826
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'


LOGIN_URL = 'account_login'
LOGIN_REDIRECT_URL = 'list_index'
LOGOUT_URL = 'account_logout'


# If you enable internal authentication, this is the address that the emails
# will appear to be coming from. Make sure you set a valid domain name,
# otherwise the emails may get rejected.
# https://docs.djangoproject.com/en/1.8/ref/settings/#default-from-email
# DEFAULT_FROM_EMAIL = "mailing-lists@you-domain.org"
DEFAULT_FROM_EMAIL = 'mailman@mailman.example.com'

# If you enable email reporting for error messages, this is where those emails
# will appear to be coming from. Make sure you set a valid domain name,
# otherwise the emails may get rejected.
# https://docs.djangoproject.com/en/1.8/ref/settings/#std:setting-SERVER_EMAIL
# SERVER_EMAIL = 'root@your-domain.org'
SERVER_EMAIL = 'mailman@mailman.example.com'

# Change this when you have a real email backend
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25


# Compatibility with Bootstrap 3
from django.contrib.messages import constants as messages  # flake8: noqa
MESSAGE_TAGS = {
    messages.ERROR: 'danger'
}


#
# Social auth
#
AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
    'allauth.account.auth_backends.AuthenticationBackend',
)

# Django Allauth
ACCOUNT_AUTHENTICATION_METHOD = "username_email"
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_EMAIL_VERIFICATION = "mandatory"
# You probably want https in production, but this is a dev setup file
ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https"
ACCOUNT_UNIQUE_EMAIL  = True

#SOCIALACCOUNT_PROVIDERS = {
#    'openid': {
#        'SERVERS': [
#            dict(id='yahoo',
#                 name='Yahoo',
#                 openid_url='http://me.yahoo.com'),
#        ],
#    },
#    'google': {
#        'SCOPE': ['profile', 'email'],
#        'AUTH_PARAMS': {'access_type': 'online'},
#    },
#    'facebook': {
#       'METHOD': 'oauth2',
#       'SCOPE': ['email'],
#       'FIELDS': [
#           'email',
#           'name',
#           'first_name',
#           'last_name',
#           'locale',
#           'timezone',
#           ],
#       'VERSION': 'v2.4',
#    },
#}


#
# Gravatar
# https://github.com/twaddington/django-gravatar
#
# Gravatar base url.
# GRAVATAR_URL = 'http://cdn.libravatar.org/'
# Gravatar base secure https url.
# GRAVATAR_SECURE_URL = 'https://seccdn.libravatar.org/'
# Gravatar size in pixels.
# GRAVATAR_DEFAULT_SIZE = '80'
# An image url or one of the following: 'mm', 'identicon', 'monsterid',
# 'wavatar', 'retro'.
# GRAVATAR_DEFAULT_IMAGE = 'mm'
# One of the following: 'g', 'pg', 'r', 'x'.
# GRAVATAR_DEFAULT_RATING = 'g'
# True to use https by default, False for plain http.
# GRAVATAR_DEFAULT_SECURE = True

#
# django-compressor
# https://pypi.python.org/pypi/django_compressor
#
COMPRESS_PRECOMPILERS = (
   ('text/less', 'lessc {infile} {outfile}'),
   ('text/x-scss', 'sassc -t compressed {infile} {outfile}'),
   ('text/x-sass', 'sassc -t compressed {infile} {outfile}'),
)
# On a production setup, setting COMPRESS_OFFLINE to True will bring a
# significant performance improvement, as CSS files will not need to be
# recompiled on each requests. It means running an additional "compress"
# management command after each code upgrade.
# http://django-compressor.readthedocs.io/en/latest/usage/#offline-compression
# COMPRESS_OFFLINE = True

# Needed for debug mode
# INTERNAL_IPS = ('127.0.0.1',)


#
# Full-text search engine
#
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine',
        'PATH': os.path.join(BASE_DIR, "fulltext_index"),
        # You can also use the Xapian engine, it's faster and more accurate,
        # but requires another library.
        # http://django-haystack.readthedocs.io/en/v2.4.1/installing_search_engines.html#xapian
        # Example configuration for Xapian:
        #'ENGINE': 'xapian_backend.XapianEngine'
    },
}


#
# Asynchronous tasks
#
Q_CLUSTER = {
    'timeout': 300,
    'save_limit': 100,
    'orm': 'default',
    'retry': 360,
}


# A sample logging configuration. The only tangible logging
# performed by this configuration is to send an email to
# the site admins on every HTTP 500 error when DEBUG=False.
# See http://docs.djangoproject.com/en/dev/topics/logging for
# more details on how to customize your logging configuration.
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'INFO',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'file':{
            'level': 'WARN',
            #'class': 'logging.handlers.RotatingFileHandler',
            'class': 'logging.handlers.WatchedFileHandler',
            'filename': os.path.join(BASE_DIR, 'logs', 'mailmansuite.log'),
            'formatter': 'verbose',
        },
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'simple',
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins', 'file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'django': {
            'handlers': ['file'],
            'level': 'INFO',
            'propagate': True,
        },
        'hyperkitty': {
            'handlers': ['file'],
            'level': 'DEBUG',
            'propagate': True,
        },
        'postorius': {
            'handlers': ['console', 'file'],
            'level': 'INFO',
        },
    },
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(process)d %(name)s %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    #'root': {
    #    'handlers': ['file'],
    #    'level': 'INFO',
    #},
}


# Using the cache infrastructure can significantly improve performance on a
# production setup. This is an example with a local Memcached server.
#CACHES = {
#    'default': {
#        'BACKEND': 'django.core.cache.backends.memcached.PyLibMCCache',
#        'LOCATION': '127.0.0.1:11211',
#    }
#}


# When DEBUG is True, don't actually send emails to the SMTP server, just store
# them in a directory. This way you won't accidentally spam your mailing-lists
# while you're fiddling with the code.
#if DEBUG == False:
#    EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
#    EMAIL_FILE_PATH = os.path.join(BASE_DIR, 'emails')


#
# HyperKitty-specific
#

# Only display mailing-lists from the same virtual host as the webserver
FILTER_VHOST = False
POSTORIUS_TEMPLATE_BASE_URL = 'https://lists.example.com'

try:
    from settings_local import *
except ImportError:
    pass

Are all the relevant components running on the same host? Are there other web services running under Apache and/or WSGI?

Nope, the virtual machine is dedicated to Mailman3. Mailman3 as venv, uwsgi as Webservice, Apache as Webserver, Postfix as Mailserver and PostgreSQL 10 als Database, all on the same host

What are the relevant configurations (Apache, uswgi, gunicorn, Mailman, Postorius, Django, database, DNS A/AAAA/CNAME)? This would be anything like port numbers and host names or IP addresses, user names and passwords (we don't need to know these, but you should provide obfuscated versions so we can check syntax, and of course you should check for typos).

Steve

Should be provided on the top of the mail.

At moment i think its a slow database. Because of the slow database, the Mailman API is slow and so on. We are trying now to solve the database part with postgresqltuner - maybe we will see more details then.

Stephan

On 1/26/22 00:11, Stephan Krinetzki wrote:

After we enabled the "slow query log", we got in the postgresql database the following output:

 2022-01-25 12:49:41.538 CET [16126] LOG:  unexpected EOF on client connection with an open transaction
 2022-01-25 15:34:36.047 CET [16864] LOG:  duration: 30760.028 ms  statement: DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX
2022-01-25 15:34:36.047 CET [3900] LOG:  duration: 9372.471 ms  statement: DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX
2022-01-25 16:42:07.243 CET [3900] LOG:  unexpected EOF on client connection with an open transaction

Pretty sounds like a database problem?

The unexpected EOF is because the gunicorn worker timed out and was killed.

There are two DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX transactions because the first one timed out. These came from https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/moderator.p...

The accept action is processed and ultimately we get to https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/moderator.p... and that in turn is doing the DELETE at https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/model/pending.p...

There must be some kind of database locking causing this, but I have no clue as to what it might be. If you haven't done so, I would stop all the mailman services and the database server and then start them again. I don't know if that would help, but it might.

sometimes the log (mailmansuite.log) show us this error

That too is from the gunicorn worker timing out and being killed.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 1/27/22 05:22, Stephan Krinetzki wrote:

I think there is a communication problem in the chain:

apache -> uwsgi -> mailmancore/API ->database

but i can't dig deeper there. Let me know if i should provide more details or logs

It seems that this part of the chain

apache -> uwsgi -> mailmancore/API

is not experiencing delay. It is involved with every Postorius request, most of which seem to have no issues. It is only this one piece

mailmancore/API ->database

that seems to have an issue and only with this one

DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX

query. However, like you, I have no idea as to what you could do further to debug this.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Stephen J. Turnbull wrote:

Mark Sapiro writes:

...

mailmancore/API ->database that seems to have an issue and only with this one DELETE FROM pendedkeyvalue WHERE pendedkeyvalue.id = XXX query. However, like you, I have no idea as to what you could do further to debug this. I have an idea, but it seems so obvious that presumably it's already been tried: log in to the database as mailman with pgsql and try deleting something. If that works immediately, try with mailmanclient and the mailman command line utility. Steve

Well, we didn't try this. Since we don't know what the query for deleting a held message in the database needs to look like, we left that out for now. If someone can help me there, it would be great.

Over the mailman shell it hangs as long as on the web ui (using mailman shell -l list@host and iterating over the held messages). So it is the communication Mailman API <-> Database.

On 1/29/22 05:48, Jacob Sievert via Mailman-users wrote:

Hello,

we seem to have the same problem right now.

We are still on python3.6 and postgresql 12.9

I looked at our Table pendedkeyvalue and we have roughly 35k rows in that table, is that normal or shouldn't those get cleaned up after a while?

Yes they should. This is https://gitlab.com/mailman/mailman/-/issues/257 fixed in Mailman core 3.3.5. Also in Mailman 3.3.5 is a new Task runner that runs periodic tasks, one of which is to remove orphaned pendings.

This may also be the issue for the OP in this thread if that pendedkeyvalue table is also large.

Here is a mailman shell script that will clean that up.

# Prior to Mailman 3.3.5, some tokens for user confirmations were pended 
with
# too long a lifetime.  This script removes those pendings based on when 
they
# were pended and the configured pending_request_life rather than their
# expiration.

# Also prior to Mailman 3.3.5, pended held_message tokens for email handling
# of the message were not removed when the message was handled via REST. 
  This
# script removes those pendings too.

# This is run with
# mailman shell -r delete_orphans_expireds
# after saving it as
# /opt/mailman/mm/venv/bin/delete_orphans_expireds.py

from datetime import datetime
from lazr.config import as_timedelta
from mailman.config import config
from mailman.database.transaction import transactional
from mailman.interfaces.pending import IPendings
from zope.component import getUtility

pendings = getUtility(IPendings)

def is_request(id):
     if config.db.store.execute(
         'SELECT * FROM _request WHERE id = {};'.format(id)).rowcount > 0:
         return True
     return False

then = datetime.now() - as_timedelta(config.mailman.pending_request_life)
thenm = datetime.now() - as_timedelta(config.mailman.moderator_request_life)

def fromisoformat(x):
     if hasattr(datetime, 'fromisoformat'):
         return datetime.fromisoformat(x)
     try:
         return datetime.strptime(x, '%Y-%m-%dT%H:%M:%S.%f')
     except ValueError:
         return datetime.strptime(x, '%Y-%m-%dT%H:%M:%S')

@transactional
def delete_orphans_expireds():
     count = 0
     for token, data in pendings.find(pend_type='held message'):
         if data and not is_request(data['id']):
             result = pendings.confirm(token, expunge=True)
             count += 1
     print(f'expunged {count} orphaned pended held messages')

     count = 0
     for token, data in pendings.find(pend_type='data'):
         if data and data['_mod_hold_date']:
             when = data['_mod_hold_date']
             if isinstance(when, str):
                 when = fromisoformat(when)
             if when < thenm:
                 result = pendings.confirm(token, expunge=True)
                 count += 1
     print(f'expunged {count} expired held messages')

     pends = list(pendings.find(pend_type='subscription'))
     pends += list(pendings.find(pend_type='unsubscription'))
     count = 0
     for token, values in pends:
         if values and values['token_owner'] == 'subscriber':
             when = values['when']
             if isinstance(when, str):
                 when = fromisoformat(when)
             if when < then:
                 result = pendings.confirm(token, expunge=True)
                 count += 1
     print(f'expunged {count} expired (un)subscription confirmations')

The above says to save the script at /opt/mailman/mm/venv/bin/delete_orphans_expireds.py but that path may need to be adjusted based in where Mailman's bin/ directory is in your installation.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Yes they should. This is https://gitlab.com/mailman/mailman/-/issues/257 fixed in Mailman core 3.3.5. Also in Mailman 3.3.5 is a new Task runner that runs periodic tasks, one of which is to remove orphaned pendings. Hmm our running system right now already was on 3.3.5 since a few days after release. But maybe all of those were still from before. This may also be the issue for the OP in this thread if that pendedkeyvalue table is also large.

Here is a mailman shell script that will clean that up.

# Prior to Mailman 3.3.5, some tokens for user confirmations were 
pended with
# too long a lifetime.  This script removes those pendings based on 
when they
# were pended and the configured pending_request_life rather than their
# expiration.
With the script we went down to around 16k entries in pendedkeyvalue
removed were the following:
expunged 6932 orphaned pended held messages
expunged 254 expired held messages
expunged 21 expired (un)subscription confirmations

This shows that there were a lot of things in an unfinished state to my understanding in our system. For a time this system was running with less RAM than it should have had. So maybe some of the problems came from there.

@mark Thanks for the script. Now the people with 300 messages pending for their lists can throw them all finally away.

regards

Jacob

Mark Sapiro wrote:

...

Hello, we seem to have the same problem right now. We are still on python3.6 and postgresql 12.9 I looked at our Table pendedkeyvalue and we have roughly 35k rows in that table, is that normal or shouldn't those get cleaned up after a while? Yes they should. This is https://gitlab.com/mailman/mailman/-/issues/257 fixed in Mailman core 3.3.5. Also in Mailman 3.3.5 is a new Task runner

On 1/29/22 05:48, Jacob Sievert via Mailman-users wrote: that runs periodic tasks, one of which is to remove orphaned pendings. This may also be the issue for the OP in this thread if that pendedkeyvalue table is also large.

I can confirm this. We have a total of ~580000 Rows in the pendedkeyvalue table. Since last wednsday even the hourly job quits (Out of memory exception)

Here is a mailman shell script that will clean that up. # Prior to Mailman 3.3.5, some tokens for user confirmations were pended with # too long a lifetime. This script removes those pendings based on when they # were pended and the configured pending_request_life rather than their # expiration.

# Also prior to Mailman 3.3.5, pended held_message tokens for email handling # of the message were not removed when the message was handled via REST. This # script removes those pendings too.

# This is run with # mailman shell -r delete_orphans_expireds # after saving it as # /opt/mailman/mm/venv/bin/delete_orphans_expireds.py

from datetime import datetime from lazr.config import as_timedelta from mailman.config import config from mailman.database.transaction import transactional from mailman.interfaces.pending import IPendings from zope.component import getUtility

pendings = getUtility(IPendings)

def is_request(id): if config.db.store.execute( 'SELECT * FROM _request WHERE id = {};'.format(id)).rowcount > 0: return True return False

then = datetime.now() - as_timedelta(config.mailman.pending_request_life) thenm = datetime.now() - as_timedelta(config.mailman.moderator_request_life)

def fromisoformat(x): if hasattr(datetime, 'fromisoformat'): return datetime.fromisoformat(x) try: return datetime.strptime(x, '%Y-%m-%dT%H:%M:%S.%f') except ValueError: return datetime.strptime(x, '%Y-%m-%dT%H:%M:%S')

@transactional def delete_orphans_expireds(): count = 0 for token, data in pendings.find(pend_type='held message'): if data and not is_request(data['id']): result = pendings.confirm(token, expunge=True) count += 1 print(f'expunged {count} orphaned pended held messages')

 count = 0
 for token, data in pendings.find(pend_type='data'):
     if data and data['_mod_hold_date']:
         when = data['_mod_hold_date']
         if isinstance(when, str):
             when = fromisoformat(when)
         if when < thenm:
             result = pendings.confirm(token, expunge=True)
             count += 1
 print(f'expunged {count} expired held messages')

 pends = list(pendings.find(pend_type='subscription'))
 pends += list(pendings.find(pend_type='unsubscription'))
 count = 0
 for token, values in pends:
     if values and values['token_owner'] == 'subscriber':
         when = values['when']
         if isinstance(when, str):
             when = fromisoformat(when)
         if when < then:
             result = pendings.confirm(token, expunge=True)
             count += 1
 print(f'expunged {count} expired (un)subscription confirmations')

The above says to save the script at /opt/mailman/mm/venv/bin/delete_orphans_expireds.py but that path may need to be adjusted based in where Mailman's bin/ directory is in your installation.

Thanks for the Script Mark! WIth the Script, the result is:

mailman shell -r delete_orphans_expireds
Jan 31 09:15:46 2022 (7980) Database url: postgres://mailman:XXXXXXXX@127.0.0.1/YYYYYYY
expunged 95918 orphaned pended held messages
expunged 7958 expired held messages
expunged 22372 expired (un)subscription confirmations

And the pendedkeyvalue table has significant lower entries (~ 92000) and now the process to accept or decline a held message ist a lot of faster. Maybe we will take the database to a seperate server to get more speed - but for the moment it is fast enough.

Thanks again and thanks to Jacob Sievert for the pointer to the size of the pendedkeyvalue table.