hyperkitty: "incorrect string value" for botched utf-8 encoding
by thoralf
hi,
my hyperkitty installation (3.2.1 from debian buster) started to act up
recently with
(1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for
column `mailman3web`.`hyperkitty_email`.`content` at row 1")
for every new mail that is about to be archived, i'll quote the full
error mail at the end of this mail.
i traced the issue to a slighty smiling face (aka u+1f642) in this
rather interestingly encoded email:
[…]
Content-Type: multipart/mixed;
boundary="===============2457401976140816263=="
--===============2457401976140816263==
Content-Type: multipart/alternative; boundary="5ecb6fd9_327b23c6_2042"
--5ecb6fd9_327b23c6_2042
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
kann halt NIE am mittwoch. viel spass =F0=9F=99=82
[…]
--5ecb6fd9_327b23c6_2042
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
<html xmlns=3D=22http://www.w3.org/1999/xhtml=22>
[…]
<div dir=3D=22auto=22>kann halt NIE am mittwoch. viel spass =F0=9F=99=82<=
/div>
--5ecb6fd9_327b23c6_2042--
[…]
--===============2457401976140816263==--
… which still lingers around in
/var/lib/mailman3/archives/hyperkitty/spool .
is this to be expected, or might it be a good idea to amend mailman3 /
hyperkitty to deal with this more gracefully?
thank you & with kind regards,
t.
---
Internal Server Error: /hyperkitty/api/mailman/archive
OperationalError at /hyperkitty/api/mailman/archive
(1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for
column `mailman3web`.`hyperkitty_email`.`content` at row 1")
Request Method: POST
Request URL: https://redacted
Django Version: 1.11.28
Python Executable: /usr/bin/uwsgi-core
Python Version: 3.7.3
Python Path: ['.', '', '/usr/lib/python37.zip', '/usr/lib/python3.7',
'/usr/lib/python3.7/lib-dynload',
'/usr/local/lib/python3.7/dist-packages', '/usr/lib/python3/dist-packages']
Server time: Mon, 25 May 2020 13:26:09 +0000
Installed Applications:
('hyperkitty',
'postorius',
'django_mailman3',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'django_gravatar',
'compressor',
'haystack',
'django_extensions',
'django_q',
'allauth',
'allauth.account',
'allauth.socialaccount')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'django_mailman3.middleware.TimezoneMiddleware',
'postorius.middleware.PostoriusMiddleware')
Traceback:
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py"
in execute
101. return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute
250. self.errorhandler(self, exc, value)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in
defaulterrorhandler
50. raise errorvalue
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute
247. res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _query
411. rowcount = self._do_query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _do_query
374. db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in query
292. _mysql.connection.query(self, query)
The above exception ((1366, "Incorrect string value:
'\\xF0\\x9F\\x99\\x82\\x0AA...' for column
`mailman3web`.`hyperkitty_email`.`content` at row 1")) was the direct
cause of the following exception:
File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py"
in inner
41. response = get_response(request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in
_get_response
187. response =
self.process_exception_by_middleware(e, request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in
_get_response
185. response = wrapped_callback(request,
*callback_args, **callback_kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/http.py" in
inner
40. return func(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py" in
_decorator
69. return func(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/csrf.py" in
wrapped_view
58. return view_func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py" in archive
115. add_to_list(mlist_fqdn, msg)
File "/usr/lib/python3/dist-packages/hyperkitty/lib/incoming.py" in
add_to_list
151. email.save()
File "/usr/lib/python3/dist-packages/django/db/models/base.py" in save
808. force_update=force_update,
update_fields=update_fields)
File "/usr/lib/python3/dist-packages/django/db/models/base.py" in save_base
838. updated = self._save_table(raw, cls, force_insert,
force_update, using, update_fields)
File "/usr/lib/python3/dist-packages/django/db/models/base.py" in
_save_table
924. result = self._do_insert(cls._base_manager, using,
fields, update_pk, raw)
File "/usr/lib/python3/dist-packages/django/db/models/base.py" in _do_insert
963. using=using, raw=raw)
File "/usr/lib/python3/dist-packages/django/db/models/manager.py" in
manager_method
85. return getattr(self.get_queryset(), name)(*args,
**kwargs)
File "/usr/lib/python3/dist-packages/django/db/models/query.py" in _insert
1079. return
query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py"
in execute_sql
1112. cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/utils.py" in __exit__
94. six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/lib/python3/dist-packages/django/utils/six.py" in reraise
685. raise value.with_traceback(tb)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py"
in execute
101. return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute
250. self.errorhandler(self, exc, value)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in
defaulterrorhandler
50. raise errorvalue
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in execute
247. res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _query
411. rowcount = self._do_query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py" in _do_query
374. db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py" in query
292. _mysql.connection.query(self, query)
Exception Type: OperationalError at /hyperkitty/api/mailman/archive
Exception Value: (1366, "Incorrect string value:
'\\xF0\\x9F\\x99\\x82\\x0AA...' for column
`mailman3web`.`hyperkitty_email`.`content` at row 1")
Request information:
USER: AnonymousUser
GET:
key = 'redacted'
POST:
mlist = 'redacted'
FILES:
message = <InMemoryUploadedFile: message.txt ()>
COOKIES: No cookie data
META:
CONTENT_LENGTH = '6223'
CONTENT_TYPE = 'multipart/form-data;
boundary=4cd93acbb8c0c45d7bd46fdf1d6a31fa'
CONTEXT_DOCUMENT_ROOT = '/var/www/html'
CONTEXT_PREFIX = ''
DOCUMENT_ROOT = '/var/www/html'
GATEWAY_INTERFACE = 'CGI/1.1'
HTTPS = 'on'
HTTP_ACCEPT = '*/*'
HTTP_ACCEPT_ENCODING = 'gzip, deflate'
HTTP_CONNECTION = 'keep-alive'
HTTP_HOST = 'lists.bing-bong.de'
HTTP_USER_AGENT = 'python-requests/2.21.0'
PATH = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
PATH_INFO = '/hyperkitty/api/mailman/archive'
QUERY_STRING = 'key=redacted'
REMOTE_ADDR = '78.47.218.110'
REMOTE_PORT = '36358'
REQUEST_METHOD = 'POST'
REQUEST_SCHEME = 'https'
REQUEST_URI = 'redacted'
SCRIPT_FILENAME = 'proxy:uwsgi://localhost/hyperkitty/api/mailman/archive'
SCRIPT_NAME = ''
SERVER_ADDR = '78.47.218.110'
SERVER_ADMIN = 'admin(a)bing-bong.de'
SERVER_NAME = 'lists.bing-bong.de'
SERVER_PORT = '443'
SERVER_PROTOCOL = 'HTTP/1.1'
SERVER_SIGNATURE = '<address>Apache/2.4.38 (Debian) Server at
lists.bing-bong.de Port 443</address>\n'
SERVER_SOFTWARE = 'Apache/2.4.38 (Debian)'
SSL_TLS_SNI = 'lists.bing-bong.de'
uwsgi.core = 1
uwsgi.node = b'bing-bong'
uwsgi.version = b'2.0.18-debian'
wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'>
wsgi.file_wrapper = ''
wsgi.input = <uwsgi._Input object at 0x7f2ae6af2d68>
wsgi.multiprocess = False
wsgi.multithread = True
wsgi.run_once = False
wsgi.url_scheme = 'https'
wsgi.version = '(1, 0)'
Settings:
Using settings module settings
ABSOLUTE_URL_OVERRIDES = {}
ACCOUNT_AUTHENTICATION_METHOD = 'username_email'
ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'https'
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_EMAIL_VERIFICATION = 'mandatory'
ACCOUNT_UNIQUE_EMAIL = True
ADMINS = "(('Mailman Suite Admin', 'mm3admin(a)bing-bong.de'),)"
ALLOWED_HOSTS = ['*']
APPEND_SLASH = True
AUTHENTICATION_BACKENDS = "('django.contrib.auth.backends.ModelBackend',
'allauth.account.auth_backends.AuthenticationBackend')"
AUTH_PASSWORD_VALIDATORS = '********************'
AUTH_USER_MODEL = 'auth.User'
BASE_DIR = '/usr/share/mailman3-web'
CACHES = {'default': {'BACKEND':
'django.core.cache.backends.locmem.LocMemCache'}}
CACHE_MIDDLEWARE_ALIAS = 'default'
CACHE_MIDDLEWARE_KEY_PREFIX = '********************'
CACHE_MIDDLEWARE_SECONDS = 600
COMPRESS_CACHEABLE_PRECOMPILERS = '()'
COMPRESS_CACHE_BACKEND = 'default'
COMPRESS_CACHE_KEY_FUNCTION = '********************'
COMPRESS_CLEAN_CSS_ARGUMENTS = ''
COMPRESS_CLEAN_CSS_BINARY = 'cleancss'
COMPRESS_CLOSURE_COMPILER_ARGUMENTS = ''
COMPRESS_CLOSURE_COMPILER_BINARY = 'java -jar compiler.jar'
COMPRESS_CSS_COMPRESSOR = 'compressor.css.CssCompressor'
COMPRESS_CSS_FILTERS = ['compressor.filters.css_default.CssAbsoluteFilter']
COMPRESS_CSS_HASHING_METHOD = 'mtime'
COMPRESS_DATA_URI_MAX_SIZE = 1024
COMPRESS_DEBUG_TOGGLE = None
COMPRESS_ENABLED = True
COMPRESS_JINJA2_GET_ENVIRONMENT = <function
CompressorConf.JINJA2_GET_ENVIRONMENT at 0x7f2ae8341f28>
COMPRESS_JS_COMPRESSOR = 'compressor.js.JsCompressor'
COMPRESS_JS_FILTERS = ['compressor.filters.jsmin.JSMinFilter']
COMPRESS_MINT_DELAY = 30
COMPRESS_MTIME_DELAY = 10
COMPRESS_OFFLINE = True
COMPRESS_OFFLINE_CONTEXT = {'STATIC_URL': '/mailman3/static/'}
COMPRESS_OFFLINE_MANIFEST = 'manifest.json'
COMPRESS_OFFLINE_TIMEOUT = 31536000
COMPRESS_OUTPUT_DIR = 'CACHE'
COMPRESS_PARSER = 'compressor.parser.AutoSelectParser'
COMPRESS_PRECOMPILERS = "(('text/less', 'lessc {infile} {outfile}'),
('text/x-scss', 'sassc -t compressed {infile} {outfile}'),
('text/x-sass', 'sassc -t compressed {infile} {outfile}'))"
COMPRESS_REBUILD_TIMEOUT = 2592000
COMPRESS_ROOT = '/var/lib/mailman3/web/static'
COMPRESS_STORAGE = 'compressor.storage.CompressorFileStorage'
COMPRESS_TEMPLATE_FILTER_CONTEXT = {'STATIC_URL': '/mailman3/static/'}
COMPRESS_URL = '/mailman3/static/'
COMPRESS_URL_PLACEHOLDER = '/__compressor_url_placeholder__/'
COMPRESS_VERBOSE = False
COMPRESS_YUGLIFY_BINARY = 'yuglify'
COMPRESS_YUGLIFY_CSS_ARGUMENTS = '--terminal'
COMPRESS_YUGLIFY_JS_ARGUMENTS = '--terminal'
COMPRESS_YUI_BINARY = 'java -jar yuicompressor.jar'
COMPRESS_YUI_CSS_ARGUMENTS = ''
COMPRESS_YUI_JS_ARGUMENTS = ''
CSRF_COOKIE_AGE = 31449600
CSRF_COOKIE_DOMAIN = None
CSRF_COOKIE_HTTPONLY = False
CSRF_COOKIE_NAME = 'csrftoken'
CSRF_COOKIE_PATH = '/'
CSRF_COOKIE_SECURE = False
CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure'
CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN'
CSRF_TRUSTED_ORIGINS = []
CSRF_USE_SESSIONS = False
DATABASES = {'default': {'ENGINE': 'django.db.backends.mysql', 'NAME':
'mailman3web', 'USER': 'mailman3web', 'PASSWORD':
'********************', 'HOST': 'localhost', 'PORT': '', 'OPTIONS':
{'init_command': "SET sql_mode='STRICT_TRANS_TABLES'"},
'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0,
'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'NAME':
None, 'MIRROR': None}}}
DATABASE_ROUTERS = []
DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440
DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000
DATETIME_FORMAT = 'N j, Y, P'
DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f',
'%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y
%H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S',
'%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y']
DATE_FORMAT = 'N j, Y'
DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y',
'%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B
%Y', '%d %B, %Y']
DEBUG = False
DEBUG_PROPAGATE_EXCEPTIONS = False
DECIMAL_SEPARATOR = '.'
DEFAULT_CHARSET = 'utf-8'
DEFAULT_CONTENT_TYPE = 'text/html'
DEFAULT_EXCEPTION_REPORTER_FILTER =
'django.views.debug.SafeExceptionReporterFilter'
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
DEFAULT_FROM_EMAIL = 'postorius(a)bing-bong.de'
DEFAULT_INDEX_TABLESPACE = ''
DEFAULT_TABLESPACE = ''
DISALLOWED_USER_AGENTS = []
EMAILNAME = 'bing-bong.de'
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_HOST_PASSWORD = '********************'
EMAIL_HOST_USER = ''
EMAIL_PORT = 25
EMAIL_SSL_CERTFILE = None
EMAIL_SSL_KEYFILE = '********************'
EMAIL_SUBJECT_PREFIX = '[Django] '
EMAIL_TIMEOUT = None
EMAIL_USE_LOCALTIME = False
EMAIL_USE_SSL = False
EMAIL_USE_TLS = False
FILE_CHARSET = 'utf-8'
FILE_UPLOAD_DIRECTORY_PERMISSIONS = None
FILE_UPLOAD_HANDLERS =
['django.core.files.uploadhandler.MemoryFileUploadHandler',
'django.core.files.uploadhandler.TemporaryFileUploadHandler']
FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440
FILE_UPLOAD_PERMISSIONS = None
FILE_UPLOAD_TEMP_DIR = None
FILTER_VHOST = False
FIRST_DAY_OF_WEEK = 0
FIXTURE_DIRS = []
FORCE_SCRIPT_NAME = None
FORMAT_MODULE_PATH = None
FORM_RENDERER = 'django.forms.renderers.DjangoTemplates'
HAYSTACK_CONNECTIONS = {'default': {'ENGINE':
'haystack.backends.whoosh_backend.WhooshEngine', 'PATH':
'/var/lib/mailman3/web/fulltext_index'}}
HOSTNAME = 'localhost.local'
IGNORABLE_404_URLS = []
INSTALLED_APPS = "('hyperkitty', 'postorius', 'django_mailman3',
'django.contrib.admin', 'django.contrib.auth',
'django.contrib.contenttypes', 'django.contrib.sessions',
'django.contrib.sites', 'django.contrib.messages',
'django.contrib.staticfiles', 'rest_framework', 'django_gravatar',
'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth',
'allauth.account', 'allauth.socialaccount')"
INTERNAL_IPS = []
LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'),
('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn',
'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'),
('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'),
('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au',
'Australian English'), ('en-gb', 'British English'), ('eo',
'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'),
('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni',
'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et',
'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'),
('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish
Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr',
'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('ia',
'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is',
'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'),
('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'),
('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk',
'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr',
'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne',
'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os',
'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'),
('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru',
'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'),
('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'),
('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'),
('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk',
'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans',
'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')]
LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur']
LANGUAGE_CODE = 'en-us'
LANGUAGE_COOKIE_AGE = None
LANGUAGE_COOKIE_DOMAIN = None
LANGUAGE_COOKIE_NAME = 'django_language'
LANGUAGE_COOKIE_PATH = '/'
LOCALE_PATHS = []
LOGGING = {'version': 1, 'disable_existing_loggers': False, 'filters':
{'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'}},
'handlers': {'mail_admins': {'level': 'ERROR', 'filters':
['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler'},
'file': {'level': 'INFO', 'class':
'logging.handlers.RotatingFileHandler', 'filename':
'/var/log/mailman3/web/mailman-web.log', 'formatter': 'verbose'},
'console': {'class': 'logging.StreamHandler', 'formatter': 'simple'}},
'loggers': {'django.request': {'handlers': ['mail_admins', 'file'],
'level': 'INFO', 'propagate': True}, 'django': {'handlers': ['file'],
'level': 'INFO', 'propagate': True}, 'hyperkitty': {'handlers':
['file'], 'level': 'INFO', 'propagate': True}, 'postorius': {'handlers':
['file'], 'level': 'INFO', 'propagate': True}}, 'formatters':
{'verbose': {'format': '%(levelname)s %(asctime)s %(process)d %(name)s
%(message)s'}, 'simple': {'format': '%(levelname)s %(message)s'}}}
LOGGING_CONFIG = 'logging.config.dictConfig'
LOGIN_REDIRECT_URL = 'list_index'
LOGIN_URL = 'account_login'
LOGOUT_REDIRECT_URL = None
LOGOUT_URL = 'account_logout'
MAILMAN_ARCHIVER_FROM = "('127.0.0.1', '::1', '78.47.218.110',
'2a01:4f8:c17:567::2/64')"
MAILMAN_ARCHIVER_KEY = '********************'
MAILMAN_REST_API_PASS = '********************'
MAILMAN_REST_API_URL = '********************'
MAILMAN_REST_API_USER = '********************'
MANAGERS = []
MEDIA_ROOT = ''
MEDIA_URL = ''
MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage'
MESSAGE_TAGS = {40: 'danger'}
MIDDLEWARE = "('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'django_mailman3.middleware.TimezoneMiddleware',
'postorius.middleware.PostoriusMiddleware')"
MIDDLEWARE_CLASSES = ['django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware']
MIGRATION_MODULES = {}
MONTH_DAY_FORMAT = 'F j'
NUMBER_GROUPING = 0
PASSWORD_HASHERS = '********************'
PASSWORD_RESET_TIMEOUT_DAYS = '********************'
POSTORIUS_TEMPLATE_BASE_URL = 'http://lists.bing-bong.de/mailman3/'
PREPEND_WWW = False
Q_CLUSTER = {'timeout': 300, 'save_limit': 100, 'orm': 'default', 'poll': 5}
ROOT_URLCONF = 'urls'
SECRET_KEY = '********************'
SECURE_BROWSER_XSS_FILTER = False
SECURE_CONTENT_TYPE_NOSNIFF = False
SECURE_HSTS_INCLUDE_SUBDOMAINS = False
SECURE_HSTS_PRELOAD = False
SECURE_HSTS_SECONDS = 0
SECURE_PROXY_SSL_HEADER = None
SECURE_REDIRECT_EXEMPT = []
SECURE_SSL_HOST = None
SECURE_SSL_REDIRECT = False
SERVER_EMAIL = 'root(a)bing-bong.de'
SESSION_CACHE_ALIAS = 'default'
SESSION_COOKIE_AGE = 1209600
SESSION_COOKIE_DOMAIN = None
SESSION_COOKIE_HTTPONLY = True
SESSION_COOKIE_NAME = 'sessionid'
SESSION_COOKIE_PATH = '/'
SESSION_COOKIE_SECURE = False
SESSION_ENGINE = 'django.contrib.sessions.backends.db'
SESSION_EXPIRE_AT_BROWSER_CLOSE = False
SESSION_FILE_PATH = None
SESSION_SAVE_EVERY_REQUEST = False
SESSION_SERIALIZER = 'django.contrib.sessions.serializers.PickleSerializer'
SETTINGS_MODULE = 'settings'
SHORT_DATETIME_FORMAT = 'm/d/Y P'
SHORT_DATE_FORMAT = 'm/d/Y'
SIGNING_BACKEND = 'django.core.signing.TimestampSigner'
SILENCED_SYSTEM_CHECKS = []
SITE_ID = 1
SOCIALACCOUNT_PROVIDERS = {}
STATICFILES_DIRS = '()'
STATICFILES_FINDERS =
"('django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder',
'compressor.finders.CompressorFinder')"
STATICFILES_STORAGE =
'django.contrib.staticfiles.storage.StaticFilesStorage'
STATIC_ROOT = '/var/lib/mailman3/web/static'
STATIC_URL = '/mailman3/static/'
TEMPLATES = [{'BACKEND':
'django.template.backends.django.DjangoTemplates', 'DIRS': [],
'APP_DIRS': True, 'OPTIONS': {'context_processors':
['django.template.context_processors.debug',
'django.template.context_processors.i18n',
'django.template.context_processors.media',
'django.template.context_processors.static',
'django.template.context_processors.tz',
'django.template.context_processors.csrf',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
'django_mailman3.context_processors.common',
'hyperkitty.context_processors.common',
'postorius.context_processors.postorius']}}]
TEST_NON_SERIALIZED_APPS = []
TEST_RUNNER = 'django.test.runner.DiscoverRunner'
THOUSAND_SEPARATOR = ','
TIME_FORMAT = 'P'
TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M']
TIME_ZONE = 'UTC'
USE_ETAGS = False
USE_I18N = True
USE_L10N = True
USE_THOUSAND_SEPARATOR = False
USE_TZ = True
USE_X_FORWARDED_HOST = True
USE_X_FORWARDED_PORT = False
WSGI_APPLICATION = 'wsgi.application'
X_FRAME_OPTIONS = 'SAMEORIGIN'
YEAR_MONTH_FORMAT = 'F Y'
5 years, 9 months
Re: Can't add emails to accept_these_nonmembers
by Mark Sapiro
On 2/26/26 13:20, Ibiam Chihurumnaya via Mailman-users wrote:
>
> I'm doing it in `mailman shell`, list_name is a List object, and it seems to have a `settings`
> attribute as `list_name.settings` returns
>
> {'acceptable_aliases': [], ...}
I would really like to know what Mailman version this is, and how it was
installed? In my case with Mailman core from the head of the GitLab branch:
```
$ mailman shell -l list.example.com
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
The variable 'm' is the list.example.com mailing list
>>> m.settings
Traceback (most recent call last):
File "<console>", line 1, in <module>
AttributeError: 'MailingList' object has no attribute 'settings'
>>> m.settings['accept_these_nonmembers']
Traceback (most recent call last):
File "<console>", line 1, in <module>
AttributeError: 'MailingList' object has no attribute 'settings'
>>>
```
Presumably in your case, you could do
```
list_name.settings['accept_these_nonmembers'].append('user(a)example.com')
or
list_name.settings['accept_these_nonmembers'].extend(['user1(a)example.com',
'user2(a)example.com', 'user3(a)example.com'])
```
but as I've said previously, `accept_these_nonmembers` is not the
preferred way to do this. Something like
```
$ mailman addmembers --delivery disabled FILE LIST
```
where FILE contains the addresses to add, one per line and LIST is the
list name would add the addresses as members with delivery disabled. To
add them as nonmembers with moderation_action Defer:
```
$ mailman shell -l list.example.com
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
The variable 'm' is the list.example.com mailing list
>>> from mailman.app.membership import add_member
>>> addresses = ('user1(a)example.com', 'user2(a)example.com',
... 'user3(a)example.com')
>>> for addr in addresses:
... rr = RequestRecord(addr)
... nm = add_member(m, rr, role=MemberRole.nonmember)
... nm.moderation_action = Action.defer
...
>>>commit()
```
Of course, all of this can be done, perhaps more easily, using the
Postorius web UI.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
3 weeks, 5 days
Re: Emergency Moderation
by Allan Hansen
> On May 24, 2020, at 11:34 , Mark Sapiro <mark(a)msapiro.net> wrote:
>
> On 5/24/20 10:59 AM, Allan Hansen wrote:
>> In the MM3 settings there is an option to ‘Hold for moderation’ when members or non-members post to the list.
>> I use that for the same cases where in MM2 I switched a list to ‘Emergency Moderation.’
>
>
> If you imported your MM 2.1 lists with `mailman import21`, `emergency`
> was set appropriately for the imported list and should be working to
> hold posts regardless of other settings.
Not sure what is the difference between ‘emergency moderation’ and ‘Hold for moderation.’ It appears redundant.
>
>
>> My own current top-of-the-list issues:
>>
>> A bug: Moderators can’t see/edit the list of members even when allowed to by list setting.
>
> The visibility issue is <https://gitlab.com/mailman/postorius/-/issues/369>.
>
> Editing membership beyond what could be done via the admindb interface
> (i.e. clearing a moderated poster's mod flag) was never allowed for a
> moderator in MM 2.1.
>
> In MM 3, a moderator can set a member's moderation when handling a held
> post. Do you think a non-owner moderator should be able to make other
> changes to a member?
I can only speak for my own application of MM3:
Desiring consistency between the lists, I don’t want to turn all the moderators into owners who can change list settings.
Many of my moderators keep asking me to see the list of members and to subscribe and remove members. I end up
servicing those requests. So...
a. Moderators should be able to unsubscribe and subscribe members.
b. Moderators should be able to set the member’s moderation status.
c. Moderators should not be able to see/modify any other member settings.
>
>
>> A feature request: Member changing email address on all lists he/she belongs to, without involving moderators.
>
>
> This is <https://gitlab.com/mailman/postorius/-/issues/399> and we're
> working on it.
Awesome!!!!
Thank you,
Allan
>
> --
> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> San Francisco Bay Area, California better use your sense - B. Dylan
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
5 years, 9 months
Re: Removing a mail addresses and users
by Abhijith PA
Hi Mark,
On 11/06/20 1:16 am, Mark Sapiro wrote:
> On 6/10/20 10:23 AM, Abhijith PA via Mailman-users wrote:
>> Hello.
>>
>> I tried to remove random email address signing ups (which I signed up
>> while testing) via admin panel. After removing I tried to sign up again,
>> but I am getting,
>>
>> 'A user is already registered with this e-mail address'
>
> You removed the user's subscription from the list, but you didn't remove
> the user from the system.
>
> Mailman3 has a concept of `user` which didn't exist in Mailman 2.1.
> User's have addresses and a user or one of the user's addresses can
> belong to a list with one or more roles (non-member, member, moderator
> or owner)
In my case these user signups were just to test postorious. These mail
accounts didn't subscribed to anything.
>> Combing through the database, I found auth_user table in mailman3web db
>> still contain those mail address. How to wipe email address entirely
>> from the db so I can signup again without the 'forget password' method
>
> There are a couple of things. If you log in to the web UI as a django
> superuser, you can go to the django admin UI -> Users section and delete
> the User, but I think that will only delete the web user and not delete
> the user from Mailman core.
>
> You can also log in to the web UI as the user and go to
> <https://lists.mailman3.org/user-profile/delete>, but that again only
> deletes the web user, and if you could do that, you could just subscribe
> once you're logged in, so I assume that's what you want to avoid.
>
> I don't think there is a way for a user as opposed to a site admin to
> delete her user record from Mailman core and even for a site admin,
> there's no way in Postorius to delete a user from core. It has to be
> done via REST
> <https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/…>
> or `mailman shell`.
>
Thanks for the link, let me try that out.
--abhijith
5 years, 9 months
Re: Confirmation emails aren't sent to users.
by Ibiam Chihurumnaya
Stephen J. Turnbull wrote:
> Ibiam Chihurumnaya via Mailman-users writes:
> > Mark Sapiro wrote:
> > > OK, this is Django. The default EMAIL_BACKEND from
> > mailman_web/settings/base.py is
> > django.core.mail.backends.smtp.EmailBackend and unless you've changed
> > that or added overrides various other defaults are
> > EMAIL_HOST = "localhost"
> > EMAIL_PORT = 25
> > EMAIL_HOST_USER = ""
> > EMAIL_HOST_PASSWORD = ""
> > EMAIL_USE_TLS = False
> > EMAIL_USE_SSL = False
> > EMAIL_SSL_CERTFILE = None
> > EMAIL_SSL_KEYFILE = None
> > Are those appropriate? Do you have an outgoing MTA listening on
> > localhost:25? What's in it's logs?
> > I don't have any of those set, didn't realize I needed to have them
> > set, the docs didn't make it obvious too.
> > Those defaults *are* set in the mailman_web settings files. You can
> override them in /etc/mailman3/settings.py, but often you don't need
> to.
Makes sense, I didn't think I needed to.
Couldn't see them with mailman conf though.
> > I also didn't notice them in schema.cfg in mailman.
> > This is for Django, whose configuration is in
> /etc/mailman3/settings.py, with defaults set in the mailmanweb package
> in your venv's site-package in the files in .../mailman_web/settings/.
> > smtp_port is currently set to 25, and I do have an outgoing MTA
> > listening on localhost:25.
> > If Mailman can distribute posts that way, Django should be able to
> send mail with the default settings.
> > Nothing shows up in the MTA logs about this particular email.
> > Not sure what to say about that. I'm not sure Postorius or Django log
> outgoing mail messages but I think they use the mailmanweb.log file by
> default if they log mail at all. (This is separate from the MTA logs
> which should record any connections from localhost and mail submitted.)
Is there any way I could get more logs, I had set Debug = True in settings.py a while back,
but can't see anything more.
> --
> GNU Mailman consultant (installation, migration, customization)
> Sirius Open Source https://www.siriusopensource.com/
> Software systems consulting in Europe, North America, and Japan
1 month, 1 week
Run 'manage.py makemigrations' to make new migrations
by christopher.claus＠tgcamberg1848.de
Hello together,
at startup of my docker-container with mailman 0.4.3 i got some warnings to apply some data-model-migrations. I´m very unfamilar with python and django and be not sure how to handle it. Are these warnings from mailman and should i migrate the data-model (and how)? Or should i ignore these message for keeping mailman in a safe state?
Best regards,
chrclaus
System check identified some issues:
WARNINGS:
django_mailman3.MailDomain: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the DjangoMailman3Config.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
django_mailman3.Profile: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the DjangoMailman3Config.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Attachment: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Email: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Favorite: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.LastView: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.MailingList: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Profile: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Tag: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Tagging: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Thread: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.ThreadCategory: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Vote: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the HyperKittyConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
postorius.EmailTemplate: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the PostoriusConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Operations to perform:
Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount
Running migrations:
No migrations to apply.
Your models in app(s): 'postorius' have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
3 years, 8 months
UnicodeEncodeError: 'ascii' codec can't encode character
by Marvin Gülker
Hi,
I've accidentally sent from an unregisted address to one of my lists.
Mailman correctly held the message back for moderation, but when I go to
the page for held messages in postorios, I receive a 500 error. I've
included an except from the log below. To me, it looks as if it errors
on processing my last name (which has a non-ascii character in it).
This is Postorius 1.1.2 on Ubuntu 18.04 LTS from the repos. I've
searched on the tracker for the problem and found one ticket[1], but it
has something with logging so I suppose this is a different problem.
Ideas? And how do I discard the message without access to the held
messages page?
Python version is 3.6.8.
Log excerpt:
Dec 09 13:21:04 2019 (3090) REST request handler error:
Traceback (most recent call last):
File "/usr/lib/python3.6/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/python3/dist-packages/mailman/database/transaction.py", line 50, in wrapper
rtn = function(*args, **kws)
File "/usr/lib/python3/dist-packages/mailman/rest/wsgiapp.py", line 214, in __call__
return super().__call__(environ, start_response)
File "falcon/api.py", line 215, in falcon.api.API.__call__ (falcon/api.c:2872)
File "falcon/api.py", line 189, in falcon.api.API.__call__ (falcon/api.c:2419)
File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 167, in on_get
resource = self._make_collection(request)
File "/usr/lib/python3/dist-packages/mailman/rest/helpers.py", line 159, in _make_collection
for resource in collection]
File "/usr/lib/python3/dist-packages/mailman/rest/helpers.py", line 159, in <listcomp>
for resource in collection]
File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 157, in _resource_as_dict
resource = self._make_resource(request.id)
File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 78, in _make_resource
resource['msg'] = msg.as_string()
File "/usr/lib/python3.6/email/message.py", line 158, in as_string
g.flatten(self, unixfrom=unixfrom)
File "/usr/lib/python3.6/email/generator.py", line 116, in flatten
self._write(msg)
File "/usr/lib/python3.6/email/generator.py", line 181, in _write
self._dispatch(msg)
File "/usr/lib/python3.6/email/generator.py", line 214, in _dispatch
meth(msg)
File "/usr/lib/python3.6/email/generator.py", line 243, in _handle_text
msg.set_payload(payload, charset)
File "/usr/lib/python3.6/email/message.py", line 315, in set_payload
payload = payload.encode(charset.output_charset)
UnicodeEncodeError: 'ascii' codec can't encode character '\ufffd' in position 269: ordinal not in range(128)
Dec 09 13:21:04 2019 (3090) 127.0.0.1 - - "GET /3.0/lists/tsc-devel(a)lists.secretchronicles.org/held?count=10&page=1 HTTP/1.1" 500 59
--
Blog: https://mg.guelker.eu
6 years, 3 months
Re: Turn off social logins?
by Stephen J. Turnbull
Note: I've changed the order of sentences from Torge's post to fit my
responses.
Torge Riedel writes:
> I do agree, same here on my side. Lot's of my users are very
> "sensitive" to social media and don't want see them here.
This is the first I've heard of this. Obviously it's fairly
widespread; *please* speak up if anyone have similar issues that we
haven't addressed. AFAIK all of the currently active Mailman
developers believe that social auth is a GoodThang[tm], so we're
unlikely to DTRT as you see it without your help.
> I need to migrate from mailman2 to mailman3 and was wondering why
> social accounts are enabled by default
By default Mailman 3 is social media: you have a profile, you can be
searched in the indicies of the archives, and so on. The large auth
providers provide more secure authentication, and a lot of convenience
for users who have such accounts already. They also take some
administrative burden off the list and site managers when people lose
their passwords and forget what their subscription address is, and
similar scenarios. Clearly, these are not universally-valued
features, but I think that they justify the current defaults.
> and are difficult to disable.
They're easy enough to disable (easy to recognize and just add a hash
character in front), since you have to edit settings_local.py to
install anyway. If you're using a packaged version and the package
configuration utility doesn't handle it, there's nothing we can do
about it. The distro will have to deal with that.
It should be better documented, I imagine (haven't checked yet).
> I propose something like an additional setting listing the enabled
> social accounts.
Do you mean in the Postorius administration interface? If so, do you
want it by-site, by-domain, or by-list? ("You" is everybody who wants
to disable social auth, not just Torge!)
If you mean in settings_local.py, I suggested something similar
earlier. It's not obvious it would be easy to do (sometimes these
things are order-dependent, though that's bad practice).
> And if the admin is setting this to an empty value in the
> settings_local.py everything is disabled
Because the settings are a Python module, this is the way
settings_local.py works anyway. That's why Mark suggested editing
INSTALLED_APPS.
7 years, 1 month
Re: moving lists to another server
by Odhiambo Washington
On Mon, Nov 20, 2023 at 8:53 PM Mark Sapiro <mark(a)msapiro.net> wrote:
> On 11/20/23 00:58, Odhiambo Washington wrote:
> >
> > What is the command to export the archive as a mbox from the DB?
> > I can't figure out which option it is by looking at 'mailman-web -h'.
>
> Normally you do this on the old server via the HyperKitty web UI which
> has download buttons which do things like
>
>
> https://example.com/archives/list/list@example.com/export/list@example.com.…
>
> or for a partial archive
>
>
> https://example.com/archives/list/list@example.com/export/list@example.com-…
>
> However, in your case I don't think you have the old installation working
> any more so the above won't help.
>
I had a dump of the old installation actually - both databases.
The above has actually helped and I obtained the archive.
> > BTW, how will ' mailman addmembers' treat a file containing:
> > ```
> > Firstname Lastname <email@address>
> > Onlyname <email2@address>
> > ```
>
> Correctly. It will import email@address with display name = 'Firstname
> Lastname' and email2@address with display name = 'Onlyname'.
>
Super!
> > I have opted for the simple way to do this:
> > 1. Create the site in Django Admin and create/configure the list in
> > Postorius
> > 2. Add the members from the file dump, with delivery set to regular.
> > 3. Send the members an email informing them that they need to set their
> > delivery preferences.
> >
> > I am stuck on the issue of exporting the archives to mbox file as already
> > stated above.
>
> Perhaps you can export the mbox via the web UI on the old server or
> maybe temporarily create a second mailman-web on your server with the
> old mailman-web database and use that web UI.
That's exactly how I did it and got the archives.
> Otherwise, if you have a working mailman-web that can access this
> database, you could try
> something like
> ```
> $ mailman-web shell
>
> >>> from hyperkitty.lib.compat import get_list_by_name
> >>> mlist = get_list_by_name('list_name_only', 'list_domain')
> >>> with open('output.mbox', 'wb') as fp:
> ... for email in mlist.emails.order_by("archived_date").all():
> ... fp.write(email.as_bytes() + b'\n')
> ...
> >>>
> ```
>
I tested the above out of curiosity. It failed. There are 9 lists in the
setup, but I am only interested in 1.
```
(venv) mailman@debian12:~$ mailman-web shell
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.17.2 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from hyperkitty.lib.compat import get_list_by_name
In [2]: mlist = get_list_by_name('list_name_only', 'list_domain')
---------------------------------------------------------------------------
Http404 Traceback (most recent call last)
Cell In[2], line 1
----> 1 mlist = get_list_by_name('list_name_only', 'list_domain')
File ~/venv/lib/python3.11/site-packages/hyperkitty/lib/compat.py:37, in
get_list_by_name(list_name, domain)
35 matching =
list(MailingList.objects.filter(name__startswith=list_name+"@"))
36 if len(matching) == 0: # no candidate found
---> 37 raise Http404("No archived mailinglist by that name")
38 if len(matching) == 1: # only one candidate
39 return matching[0]
Http404: No archived mailinglist by that name
```
Was I supposed to substitute something for the real listname of the list I
wanted?
Another thing - totally unrelated - that surprised me:
While using the commented-out block below, I could not click and get
anything beyond http://http://list.name/mailman3 as anything to do with
/postorius/xxxxx gave "Page not found" error.
However, with the second config block that uses the UNIX socket, I am able
to access everything
```
#<IfModule mod_proxy.c>
# ProxyPreserveHost On
# ProxyPass "/mailman3" "http://127.0.0.1:8010/mailman3"
# ProxyPass "/archives" "http://127.0.0.1:8010/archives"
# ProxyPass "/accounts" "http://127.0.0.1:8010/accounts"
# ProxyPass "/admin" "http://127.0.0.1:8010/admin"
# ProxyPass "/user-profile" "http://127.0.0.1:8010/user-profile"
# </IfModule>
ProxyPassMatch ^/static/ !
ProxyPassMatch ^/favicon.ico !
ProxyPass / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
ProxyPassReverse / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
```
Why is it that the UNIX socket option worked flawlessly while the other
option failed???
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
[How to ask smart questions:
http://www.catb.org/~esr/faqs/smart-questions.html]
2 years, 4 months
Re: Cross-List and Domain based sender whitelisting ( accept_these_nonmembers )
by Abhilash Raj
On 9/8/21 3:39 PM, Patrick Brückner wrote:
> Hi everyone,
>
> one of our servers that was running a mailman2 instance just crashed this afternoon, so we decided to take the leap and migrate to 3, and I have to say it worked pretty well, the lists seem to be back.
>
> What I can't seem to find is the option to whitelist senders based on a domain AND on membership on a different list.
>
> The lists are operated for a smallish university and all the professors need to be allowed to send to the students list, without being member of that list. That was possible with mailman2 by just whitelisting the university domain.
>
> Also, in the past we allowed all members of the all-students list to post to any of the dorm-lists (this is mainly used for party announcements so it is not a must have), but I wanted to ask anyways, if there was a chance to allow this feature again.
>
> According to the docs, the accept_these_nonmembers option exists (https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/…) but none of my lists have that set after the migration - the field is not even there when accessing the REST API.
>
> Do I need to enable something? Has that feature been added just recently, and we are missing out (since we are using the apt packages from ubuntu)?
You are probably on an old version of Mailman which didn't expose the
values in API and also Postorius didn't expose it in UI.
You can install a new version manually by downloading new debs from
Debian unstable repos (I don't know if Ubuntu just copies the ones from
Debian or has their own packaging, I think it is former) that have newer
built debs.
On your version, you probably would be able to set the
accept_these_nonmembers using the command line using
```
mailman withlist -l test(a)example.com
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
The variable 'm' is the test(a)example.com mailing list
>>> m.accept_these_nonmembers
['*.example.com']
>>> m.accept_these_nonmembers = [r'*.example.com']
>>> commit()
>>> quit()
```
--
thanks,
Abhilash Raj (maxking)
4 years, 6 months