- Mailman-users - lists.mailman3.org

hyperkitty: "incorrect string value" for botched utf-8 encoding
by thoralf June 20, 2020

June 20, 2020

hi, my hyperkitty installation (3.2.1 from debian buster) started to act up recently with (1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for column `mailman3web`.`hyperkitty_email`.`content` at row 1") for every new mail that is about to be archived, i'll quote the full error mail at the end of this mail. i traced the issue to a slighty smiling face (aka u+1f642) in this rather interestingly encoded email: […] Content-Type: multipart/mixed; boundary="===============2457401976140816263==" --===============2457401976140816263== Content-Type: multipart/alternative; boundary="5ecb6fd9_327b23c6_2042" --5ecb6fd9_327b23c6_2042 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline kann halt NIE am mittwoch. viel spass =F0=9F=99=82 […] --5ecb6fd9_327b23c6_2042 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline <html xmlns=3D=22http://www.w3.org/1999/xhtml=22> […] <div dir=3D=22auto=22>kann halt NIE am mittwoch. viel spass =F0=9F=99=82<= /div> --5ecb6fd9_327b23c6_2042-- […] --===============2457401976140816263==-- … which still lingers around in /var/lib/mailman3/archives/hyperkitty/spool . is this to be expected, or might it be a good idea to amend mailman3 / hyperkitty to deal with this more gracefully? thank you & with kind regards, t. --- Internal Server Error: /hyperkitty/api/mailman/archive OperationalError at /hyperkitty/api/mailman/archive (1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for column `mailman3web`.`hyperkitty_email`.`content` at row 1") Request Method: POST Request URL: https://redacted Django Version: 1.11.28 Python Executable: /usr/bin/uwsgi-core Python Version: 3.7.3 Python Path: ['.', '', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages'] Server time: Mon, 25 May 2020 13:26:09 +0000 Installed Applications: ('hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount') Installed Middleware: ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware') Traceback: File "/usr/lib/python3/dist-packages/django/db/backends/utils.py" in execute 64. return self.cursor.execute(sql, params) File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py" in execute 101. return self.cursor.execute(query, args) File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute 250. self.errorhandler(self, exc, value) File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in defaulterrorhandler 50. raise errorvalue File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute 247. res = self._query(query) File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _query 411. rowcount = self._do_query(q) File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _do_query 374. db.query(q) File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in query 292. _mysql.connection.query(self, query) The above exception ((1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for column `mailman3web`.`hyperkitty_email`.`content` at row 1")) was the direct cause of the following exception: File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py" in inner 41. response = get_response(request) File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response 187. response = self.process_exception_by_middleware(e, request) File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response 185. response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python3/dist-packages/django/views/decorators/http.py" in inner 40. return func(request, *args, **kwargs) File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py" in _decorator 69. return func(request, *args, **kwargs) File "/usr/lib/python3/dist-packages/django/views/decorators/csrf.py" in wrapped_view 58. return view_func(*args, **kwargs) File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py" in archive 115. add_to_list(mlist_fqdn, msg) File "/usr/lib/python3/dist-packages/hyperkitty/lib/incoming.py" in add_to_list 151. email.save() File "/usr/lib/python3/dist-packages/django/db/models/base.py" in save 808. force_update=force_update, update_fields=update_fields) File "/usr/lib/python3/dist-packages/django/db/models/base.py" in save_base 838. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/usr/lib/python3/dist-packages/django/db/models/base.py" in _save_table 924. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/usr/lib/python3/dist-packages/django/db/models/base.py" in _do_insert 963. using=using, raw=raw) File "/usr/lib/python3/dist-packages/django/db/models/manager.py" in manager_method 85. return getattr(self.get_queryset(), name)(*args, **kwargs) File "/usr/lib/python3/dist-packages/django/db/models/query.py" in _insert 1079. return query.get_compiler(using=using).execute_sql(return_id) File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py" in execute_sql 1112. cursor.execute(sql, params) File "/usr/lib/python3/dist-packages/django/db/backends/utils.py" in execute 64. return self.cursor.execute(sql, params) File "/usr/lib/python3/dist-packages/django/db/utils.py" in __exit__ 94. six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/lib/python3/dist-packages/django/utils/six.py" in reraise 685. raise value.with_traceback(tb) File "/usr/lib/python3/dist-packages/django/db/backends/utils.py" in execute 64. return self.cursor.execute(sql, params) File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py" in execute 101. return self.cursor.execute(query, args) File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute 250. self.errorhandler(self, exc, value) File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in defaulterrorhandler 50. raise errorvalue File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute 247. res = self._query(query) File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _query 411. rowcount = self._do_query(q) File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _do_query 374. db.query(q) File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in query 292. _mysql.connection.query(self, query) Exception Type: OperationalError at /hyperkitty/api/mailman/archive Exception Value: (1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for column `mailman3web`.`hyperkitty_email`.`content` at row 1") Request information: USER: AnonymousUser GET: key = 'redacted' POST: mlist = 'redacted' FILES: message = <InMemoryUploadedFile: message.txt ()> COOKIES: No cookie data META: CONTENT_LENGTH = '6223' CONTENT_TYPE = 'multipart/form-data; boundary=4cd93acbb8c0c45d7bd46fdf1d6a31fa' CONTEXT_DOCUMENT_ROOT = '/var/www/html' CONTEXT_PREFIX = '' DOCUMENT_ROOT = '/var/www/html' GATEWAY_INTERFACE = 'CGI/1.1' HTTPS = 'on' HTTP_ACCEPT = '*/*' HTTP_ACCEPT_ENCODING = 'gzip, deflate' HTTP_CONNECTION = 'keep-alive' HTTP_HOST = 'lists.bing-bong.de' HTTP_USER_AGENT = 'python-requests/2.21.0' PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' PATH_INFO = '/hyperkitty/api/mailman/archive' QUERY_STRING = 'key=redacted' REMOTE_ADDR = '78.47.218.110' REMOTE_PORT = '36358' REQUEST_METHOD = 'POST' REQUEST_SCHEME = 'https' REQUEST_URI = 'redacted' SCRIPT_FILENAME = 'proxy:uwsgi://localhost/hyperkitty/api/mailman/archive' SCRIPT_NAME = '' SERVER_ADDR = '78.47.218.110' SERVER_ADMIN = 'admin(a)bing-bong.de' SERVER_NAME = 'lists.bing-bong.de' SERVER_PORT = '443' SERVER_PROTOCOL = 'HTTP/1.1' SERVER_SIGNATURE = '<address>Apache/2.4.38 (Debian) Server at lists.bing-bong.de Port 443</address>\n' SERVER_SOFTWARE = 'Apache/2.4.38 (Debian)' SSL_TLS_SNI = 'lists.bing-bong.de' uwsgi.core = 1 uwsgi.node = b'bing-bong' uwsgi.version = b'2.0.18-debian' wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'> wsgi.file_wrapper = '' wsgi.input = <uwsgi._Input object at 0x7f2ae6af2d68> wsgi.multiprocess = False wsgi.multithread = True wsgi.run_once = False wsgi.url_scheme = 'https' wsgi.version = '(1, 0)' Settings: Using settings module settings ABSOLUTE_URL_OVERRIDES = {} ACCOUNT_AUTHENTICATION_METHOD = 'username_email' ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'https' ACCOUNT_EMAIL_REQUIRED = True ACCOUNT_EMAIL_VERIFICATION = 'mandatory' ACCOUNT_UNIQUE_EMAIL = True ADMINS = "(('Mailman Suite Admin', 'mm3admin(a)bing-bong.de'),)" ALLOWED_HOSTS = ['*'] APPEND_SLASH = True AUTHENTICATION_BACKENDS = "('django.contrib.auth.backends.ModelBackend', 'allauth.account.auth_backends.AuthenticationBackend')" AUTH_PASSWORD_VALIDATORS = '********************' AUTH_USER_MODEL = 'auth.User' BASE_DIR = '/usr/share/mailman3-web' CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} CACHE_MIDDLEWARE_ALIAS = 'default' CACHE_MIDDLEWARE_KEY_PREFIX = '********************' CACHE_MIDDLEWARE_SECONDS = 600 COMPRESS_CACHEABLE_PRECOMPILERS = '()' COMPRESS_CACHE_BACKEND = 'default' COMPRESS_CACHE_KEY_FUNCTION = '********************' COMPRESS_CLEAN_CSS_ARGUMENTS = '' COMPRESS_CLEAN_CSS_BINARY = 'cleancss' COMPRESS_CLOSURE_COMPILER_ARGUMENTS = '' COMPRESS_CLOSURE_COMPILER_BINARY = 'java -jar compiler.jar' COMPRESS_CSS_COMPRESSOR = 'compressor.css.CssCompressor' COMPRESS_CSS_FILTERS = ['compressor.filters.css_default.CssAbsoluteFilter'] COMPRESS_CSS_HASHING_METHOD = 'mtime' COMPRESS_DATA_URI_MAX_SIZE = 1024 COMPRESS_DEBUG_TOGGLE = None COMPRESS_ENABLED = True COMPRESS_JINJA2_GET_ENVIRONMENT = <function CompressorConf.JINJA2_GET_ENVIRONMENT at 0x7f2ae8341f28> COMPRESS_JS_COMPRESSOR = 'compressor.js.JsCompressor' COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter'] COMPRESS_MINT_DELAY = 30 COMPRESS_MTIME_DELAY = 10 COMPRESS_OFFLINE = True COMPRESS_OFFLINE_CONTEXT = {'STATIC_URL': '/mailman3/static/'} COMPRESS_OFFLINE_MANIFEST = 'manifest.json' COMPRESS_OFFLINE_TIMEOUT = 31536000 COMPRESS_OUTPUT_DIR = 'CACHE' COMPRESS_PARSER = 'compressor.parser.AutoSelectParser' COMPRESS_PRECOMPILERS = "(('text/less', 'lessc {infile} {outfile}'), ('text/x-scss', 'sassc -t compressed {infile} {outfile}'), ('text/x-sass', 'sassc -t compressed {infile} {outfile}'))" COMPRESS_REBUILD_TIMEOUT = 2592000 COMPRESS_ROOT = '/var/lib/mailman3/web/static' COMPRESS_STORAGE = 'compressor.storage.CompressorFileStorage' COMPRESS_TEMPLATE_FILTER_CONTEXT = {'STATIC_URL': '/mailman3/static/'} COMPRESS_URL = '/mailman3/static/' COMPRESS_URL_PLACEHOLDER = '/__compressor_url_placeholder__/' COMPRESS_VERBOSE = False COMPRESS_YUGLIFY_BINARY = 'yuglify' COMPRESS_YUGLIFY_CSS_ARGUMENTS = '--terminal' COMPRESS_YUGLIFY_JS_ARGUMENTS = '--terminal' COMPRESS_YUI_BINARY = 'java -jar yuicompressor.jar' COMPRESS_YUI_CSS_ARGUMENTS = '' COMPRESS_YUI_JS_ARGUMENTS = '' CSRF_COOKIE_AGE = 31449600 CSRF_COOKIE_DOMAIN = None CSRF_COOKIE_HTTPONLY = False CSRF_COOKIE_NAME = 'csrftoken' CSRF_COOKIE_PATH = '/' CSRF_COOKIE_SECURE = False CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure' CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN' CSRF_TRUSTED_ORIGINS = [] CSRF_USE_SESSIONS = False DATABASES = {'default': {'ENGINE': 'django.db.backends.mysql', 'NAME': 'mailman3web', 'USER': 'mailman3web', 'PASSWORD': '********************', 'HOST': 'localhost', 'PORT': '', 'OPTIONS': {'init_command': "SET sql_mode='STRICT_TRANS_TABLES'"}, 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'NAME': None, 'MIRROR': None}}} DATABASE_ROUTERS = [] DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440 DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000 DATETIME_FORMAT = 'N j, Y, P' DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y'] DATE_FORMAT = 'N j, Y' DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y'] DEBUG = False DEBUG_PROPAGATE_EXCEPTIONS = False DECIMAL_SEPARATOR = '.' DEFAULT_CHARSET = 'utf-8' DEFAULT_CONTENT_TYPE = 'text/html' DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter' DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' DEFAULT_FROM_EMAIL = 'postorius(a)bing-bong.de' DEFAULT_INDEX_TABLESPACE = '' DEFAULT_TABLESPACE = '' DISALLOWED_USER_AGENTS = [] EMAILNAME = 'bing-bong.de' EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_HOST_PASSWORD = '********************' EMAIL_HOST_USER = '' EMAIL_PORT = 25 EMAIL_SSL_CERTFILE = None EMAIL_SSL_KEYFILE = '********************' EMAIL_SUBJECT_PREFIX = '[Django] ' EMAIL_TIMEOUT = None EMAIL_USE_LOCALTIME = False EMAIL_USE_SSL = False EMAIL_USE_TLS = False FILE_CHARSET = 'utf-8' FILE_UPLOAD_DIRECTORY_PERMISSIONS = None FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler'] FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440 FILE_UPLOAD_PERMISSIONS = None FILE_UPLOAD_TEMP_DIR = None FILTER_VHOST = False FIRST_DAY_OF_WEEK = 0 FIXTURE_DIRS = [] FORCE_SCRIPT_NAME = None FORMAT_MODULE_PATH = None FORM_RENDERER = 'django.forms.renderers.DjangoTemplates' HAYSTACK_CONNECTIONS = {'default': {'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine', 'PATH': '/var/lib/mailman3/web/fulltext_index'}} HOSTNAME = 'localhost.local' IGNORABLE_404_URLS = [] INSTALLED_APPS = "('hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount')" INTERNAL_IPS = [] LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')] LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur'] LANGUAGE_CODE = 'en-us' LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_NAME = 'django_language' LANGUAGE_COOKIE_PATH = '/' LOCALE_PATHS = [] LOGGING = {'version': 1, 'disable_existing_loggers': False, 'filters': {'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'}}, 'handlers': {'mail_admins': {'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler'}, 'file': {'level': 'INFO', 'class': 'logging.handlers.RotatingFileHandler', 'filename': '/var/log/mailman3/web/mailman-web.log', 'formatter': 'verbose'}, 'console': {'class': 'logging.StreamHandler', 'formatter': 'simple'}}, 'loggers': {'django.request': {'handlers': ['mail_admins', 'file'], 'level': 'INFO', 'propagate': True}, 'django': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}, 'hyperkitty': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}, 'postorius': {'handlers': ['file'], 'level': 'INFO', 'propagate': True}}, 'formatters': {'verbose': {'format': '%(levelname)s %(asctime)s %(process)d %(name)s %(message)s'}, 'simple': {'format': '%(levelname)s %(message)s'}}} LOGGING_CONFIG = 'logging.config.dictConfig' LOGIN_REDIRECT_URL = 'list_index' LOGIN_URL = 'account_login' LOGOUT_REDIRECT_URL = None LOGOUT_URL = 'account_logout' MAILMAN_ARCHIVER_FROM = "('127.0.0.1', '::1', '78.47.218.110', '2a01:4f8:c17:567::2/64')" MAILMAN_ARCHIVER_KEY = '********************' MAILMAN_REST_API_PASS = '********************' MAILMAN_REST_API_URL = '********************' MAILMAN_REST_API_USER = '********************' MANAGERS = [] MEDIA_ROOT = '' MEDIA_URL = '' MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage' MESSAGE_TAGS = {40: 'danger'} MIDDLEWARE = "('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')" MIDDLEWARE_CLASSES = ['django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware'] MIGRATION_MODULES = {} MONTH_DAY_FORMAT = 'F j' NUMBER_GROUPING = 0 PASSWORD_HASHERS = '********************' PASSWORD_RESET_TIMEOUT_DAYS = '********************' POSTORIUS_TEMPLATE_BASE_URL = 'http://lists.bing-bong.de/mailman3/' PREPEND_WWW = False Q_CLUSTER = {'timeout': 300, 'save_limit': 100, 'orm': 'default', 'poll': 5} ROOT_URLCONF = 'urls' SECRET_KEY = '********************' SECURE_BROWSER_XSS_FILTER = False SECURE_CONTENT_TYPE_NOSNIFF = False SECURE_HSTS_INCLUDE_SUBDOMAINS = False SECURE_HSTS_PRELOAD = False SECURE_HSTS_SECONDS = 0 SECURE_PROXY_SSL_HEADER = None SECURE_REDIRECT_EXEMPT = [] SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False SERVER_EMAIL = 'root(a)bing-bong.de' SESSION_CACHE_ALIAS = 'default' SESSION_COOKIE_AGE = 1209600 SESSION_COOKIE_DOMAIN = None SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_NAME = 'sessionid' SESSION_COOKIE_PATH = '/' SESSION_COOKIE_SECURE = False SESSION_ENGINE = 'django.contrib.sessions.backends.db' SESSION_EXPIRE_AT_BROWSER_CLOSE = False SESSION_FILE_PATH = None SESSION_SAVE_EVERY_REQUEST = False SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer' SETTINGS_MODULE = 'settings' SHORT_DATETIME_FORMAT = 'm/d/Y P' SHORT_DATE_FORMAT = 'm/d/Y' SIGNING_BACKEND = 'django.core.signing.TimestampSigner' SILENCED_SYSTEM_CHECKS = [] SITE_ID = 1 SOCIALACCOUNT_PROVIDERS = {} STATICFILES_DIRS = '()' STATICFILES_FINDERS = "('django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'compressor.finders.CompressorFinder')" STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' STATIC_ROOT = '/var/lib/mailman3/web/static' STATIC_URL = '/mailman3/static/' TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.media', 'django.template.context_processors.static', 'django.template.context_processors.tz', 'django.template.context_processors.csrf', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'django_mailman3.context_processors.common', 'hyperkitty.context_processors.common', 'postorius.context_processors.postorius']}}] TEST_NON_SERIALIZED_APPS = [] TEST_RUNNER = 'django.test.runner.DiscoverRunner' THOUSAND_SEPARATOR = ',' TIME_FORMAT = 'P' TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] TIME_ZONE = 'UTC' USE_ETAGS = False USE_I18N = True USE_L10N = True USE_THOUSAND_SEPARATOR = False USE_TZ = True USE_X_FORWARDED_HOST = True USE_X_FORWARDED_PORT = False WSGI_APPLICATION = 'wsgi.application' X_FRAME_OPTIONS = 'SAMEORIGIN' YEAR_MONTH_FORMAT = 'F Y'

2 3

locked files problem after reboot
by mailman3＠malevil.cc June 20, 2020

June 20, 2020

hey, I have got a problem with flufl.lock in mailman3. When my server is rebooted, locked files seems to stay and cause errors so mailman3 core wont start (it is started automatically by supervisord service): PermissionError: [Errno 1] Operation not permitted Traceback (most recent call last): File "/home/user/.local/lib/python3.6/site-packages/mailman/bin/master.py", line 146, in acquire_lock lock = acquire_lock_1(force) File "/home/user/.local/lib/python3.6/site-packages/mailman/bin/master.py", line 125, in acquire_lock_1 lock.lock(timedelta(seconds=0.1)) File "/home/user/.local/lib/python3.6/site-packages/flufl/lock/_lockfile.py", line 259, in lock raise TimeOutError('Could not acquire the lock') flufl.lock._lockfile.TimeOutError: Could not acquire the lock I have to remove the files in var/locks manually to go forward. Is there a configuration error for my setup? Or is it just a not fixed bug? Greetsn, jekyl

3 4

Exim: is it possible to host mailman and exim on different hosts? Howto?
by Jörg Schulz June 20, 2020

June 20, 2020

as of the documentation, https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.h… , exim4 (and postfix similarly) need to have access to the mailman filesystem (MM3_HOME=/opt/mailman/var). If exim and MM would be on different hosts (or different docker environments), how would we need to configure that?

3 2

What database to choose for production deployment: Postgres or MySql
by pgrzywaczewski＠gmail.com June 20, 2020

June 20, 2020

Hi, What database is better for large deployment: Postgres or MySql? 'Large' means - around 15k mailing lists - 36k incoming messages per day - 270k outgoing messages per day distributed to recipients Is there preference from performance point of view? Or something else worth mentioning? Thanks, Paweł

4 4

how to delete a list member using the Linux command line
by Mike Wertheim June 19, 2020

June 19, 2020

I have Mailman 3.2.1 installed (on Debian Linux), using exim4 as the MTA. I have been trying to figure out how to use the Linux command line to add members to a list and delete members from a list, but am having trouble with deleting. Here is what works for adding a list member... % echo mike(a)foo.com | mailman members --add - mylist(a)domain.org Here is what works for viewing the list members... % mailman members mylist(a)domain.org But I'm having trouble deleting a member. Here is what I have tried that didn't work... # echo mike(a)foo.com | mailman members --delete - mylist(a)domain.org Usage: mailman members [OPTIONS] LISTSPEC Try "mailman members -h" for help. Error: no such option: --delete # echo mike(a)foo.com | mailman members --delmembers - mylist(a)domain.org Usage: mailman members [OPTIONS] LISTSPEC Try "mailman members -h" for help. Error: no such option: --delmembers What is the correct command to delete the member mike(a)foo.com from the list mylist(a)domain.org? (I ran "mailman members -h", and it mentions "add" but does not mention deleting at all.)

2 3

LTMP to Postfix problem
by Brian Carpenter June 19, 2020

June 19, 2020

On one of my mailman 3 servers, I am having a problem with Postfix failing to deliver email to Mailman. I keep seeing this in the mail.log: status=deferred (connect to 127.0.0.1[127.0.0.1]:8024: Connection refused) What is causing this and how can I fix it? -- Please let me know if you need further assistance. Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com -- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase EMWD's Community Forums http://discourse.emwd.com/

4 12

Mailman do not honor non-members moderation setting sometimes
by Alan So June 19, 2020

June 19, 2020

Some of our users reported that their list do not honor the nonmembers moderation setting sometimes. We were not able to repeat the issue consistently earlier. Luckily, we found the following way to repeat the issue: 1) Create a new list with user(a)example.com as owner. (Ordinary disussion mailing list style) 2) Change the list Settings -> Message Acceptance -> Default action to take when a non-member posts to the list: Discard 3) Add a non-member user(a)example.com and change the moderation setting to Hold for Moderation 4) Send an email to the list using user(a)example.com. The message is discarded instead of being held. The above can also be repeated with the list setting Default action to take when a non-member posts to the list: Accept immediately. The message sent from user(a)example.com is accepted instead of being held. So, it looks like that the non-members moderation setting is ignored for this case. Further study indicate that the issue did not happen if the owner is set to user(a)example.com *after* adding the non-member. It looks like that we can avoid the issue during list creation. However, the issue can also be triggered through other ways such as adding the non-member as member and remove. There are also ways to resolve the issue by removing all corresponding member, non-member, owner setting and add them back in the following order: non-member, owner. However, it does not always work and so I do not go into the detail at this moment to complicate the issue.

2 2

Affinity Mailman Is Now Beta
by Brian Carpenter June 18, 2020

June 18, 2020

Dear Mailman Admins and Users, Affinity Mailman is our new admin interface for Mailman 3 and we intend to use it to replace Postorius. It has been under development for about 3 months now and it recently reached beta. Calling For Beta Testers Now that Affinity is in beta, we are looking for list admins to help us beta test it. We are interested in getting some feedback and feature requests from beta users. If you are interested in helping us test out Affinity and get a sneak peek of the future of Mailman administration, then email me at brian_carpenter(a)emwd.com <mailto:brian_carpenter@emwd.com?subject=Affinity%20Beta&body=I%20am%20interested%20in%20Beta%20Testing%20Affinity> and I will begin the process of setting you up with an Affinity Mailman admin account and provide 2 test lists to use for beta testing purposes. While I am at it: Empathy Is Now Under Development "So what in the world is Empathy? I mean really - first Affinity and now Empathy?" That is correct. We figured if we are going to replace the default admin UI (Postorius) for Mailman 3, we might as well replace the default forum/archive UI (Hyperkitty) for Mailman 3. With Empathy, we are bringing a beautiful and user-friendly experience for those who wish to use Mailman 3 as a modern forum. We want Empathy to bring a true community like approach to the use of Mailman 3 via a browser. It will supplement the mailing list feature of Mailman 3 with an easy to use online forum. So Mailman 3 users will get the best of both worlds: a modern mailing list app AND a modern forum app. Empathy can also be used to display your list threads (archives) in an easy-to-read manner, and those threads can be categorized by tags and will be searchable. -- Please let me know if you need further assistance. Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com -- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase EMWD's Community Forums http://discourse.emwd.com/

2 1

mailman2 vs Mailman3 database usage
by Ralf Wiegand June 18, 2020

June 18, 2020

Do you have a list of the information that is stored in the DB? I don’t think it is emails, so it may be an issue of the number of lists and email addresses or users that are allowed to post or moderate. Thank You, Ralf Wiegand -- Ralf K. Wiegand -- Leidos 6909 Metro Park Dr Franconia, VA 22310 469-289-7372 (mobile) ralf.wiegand(a)tena-sda.org

1 0

Exception in the HyperKitty archiver: 'utf-8' codec can't encode characters
by Gilles Filippini June 18, 2020

June 18, 2020

Hi, I'm seeing the above error in mailman-core logs each time an email is processed. Is there something I missed in the configuration? I'm using dockerized mailman 0.3 from maxking. Thanks in advance, _g.

2 7