On 7/19/24 4:20 PM, Bryan Fields wrote:

Here's a link to the server in question, it's just a text box as we work through this. There's about 14k people on this list, so we want to get it right before cut over.

https://nanog-mm3.keekles.org/archives/list/mm2-mm3@nanog-mm3.keekles.org/

https://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/

Doh! gave the wrong URL. This is the correct list, and if you click "thread by month" you see it stops at July 2007.

-- Bryan Fields

727-409-1194 - Voice http://bryanfields.net

Sorry for the delay, I was on some travel last week.

On 7/19/24 7:30 PM, Mark Sapiro wrote:

That URL gives me as server error. What's in the Django (mailman-web) logs for that?

I did notice hitting a 500 error on occasion, sometimes it works.

When this happens, I get an email error below.

Are you running all the Django periodic jobs and is qcluster running?

Ok, the docs said that qcluster service wasn't needed under uWSGI https://docs.mailman3.org/en/latest/install/virtualenv.html#automatically-st... qcluster does appear to be running:

218916 ? Sl 0:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 218929 ? Sl 0:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940851 ? Ss 0:00 /bin/sh -c /opt/mailman/venv/bin/mailman-web qcluster 3940854 ? S 0:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940856 ? S 117:32 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940860 ? S 0:28 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940861 ? Sl 3048:11 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster

=== Email Error === Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /archives/list/nanog@nanog-mm3.keekles.org/

Internal Server Error: /archives/list/nanog@nanog-mm3.keekles.org/

NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$']

Request Method: GET Request URL: http://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/ Django Version: 4.2.13 Python Executable: /opt/mailman/venv/bin/uwsgi Python Version: 3.10.12 Python Path: ['.', '', '/etc/mailman3', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/opt/mailman/venv/lib/python3.10/site-packages'] Server time: Thu, 25 Jul 2024 18:16:17 +0000 Installed Applications: ['hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.humanize', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount'] Installed Middleware: ('allauth.account.middleware.AccountMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')

Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/lib/view_helpers.py", line 137, in inner return func(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/views/mlist.py", line 239, in overview return render(request, "hyperkitty/overview.html", context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/shortcuts.py", line 24, in render content = loader.render_to_string(template_name, context, request, using=using) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader.py", line 62, in render_to_string return template.render(context, request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/backends/django.py", line 61, in render return self.template.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 175, in render return self._render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader_tags.py", line 157, in render return compiled_parent._render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader_tags.py", line 63, in render result = block.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/templatetags/cache.py", line 53, in render value = self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader_tags.py", line 208, in render return template.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 177, in render return self._render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/defaulttags.py", line 238, in render nodelist.append(node.render_annotated(context)) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/defaulttags.py", line 238, in render nodelist.append(node.render_annotated(context)) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/defaulttags.py", line 471, in render url = reverse(view_name, args=args, kwargs=kwargs, current_app=current_app) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/base.py", line 88, in reverse return resolver._reverse_with_prefix(view, prefix, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/resolvers.py", line 828, in _reverse_with_prefix raise NoReverseMatch(msg)

Exception Type: NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Exception Value: Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$'] Raised during: hyperkitty.views.mlist.overview Request information: USER: nanogadmin

GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES: csrftoken = '********************' sessionid = '********************'

META: CSRF_COOKIE = '6F7ioB7blmJwLwKQLH0DPuAoWTW8mQGI' HTTP_ACCEPT = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' HTTP_ACCEPT_ENCODING = 'gzip, deflate, br' HTTP_ACCEPT_LANGUAGE = 'en-US,en;q=0.5' HTTP_CONNECTION = 'Keep-Alive' HTTP_COOKIE = '********************' HTTP_HOST = 'nanog-mm3.keekles.org' HTTP_SEC_FETCH_DEST = 'document' HTTP_SEC_FETCH_MODE = 'navigate' HTTP_SEC_FETCH_SITE = 'none' HTTP_SEC_FETCH_USER = '?1' HTTP_UPGRADE_INSECURE_REQUESTS = '1' HTTP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:109.0) Gecko/20100101 Firefox/115.0' HTTP_X_FORWARDED_FOR = '47.206.239.202' HTTP_X_FORWARDED_HOST = 'nanog-mm3.keekles.org' HTTP_X_FORWARDED_SERVER = 'nanog-mm3.keekles.org' PATH_INFO = '/archives/list/nanog@nanog-mm3.keekles.org/' QUERY_STRING = '' REMOTE_ADDR = '127.0.0.1' REQUEST_METHOD = 'GET' REQUEST_URI = '/archives/list/nanog@nanog-mm3.keekles.org/' SCRIPT_NAME = '' SERVER_NAME = 'nanog-mm3' SERVER_PORT = '8000' SERVER_PROTOCOL = 'HTTP/1.1' uwsgi.core = 1 uwsgi.node = b'nanog-mm3' uwsgi.version = b'2.0.26' wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'> wsgi.file_wrapper = <built-in function uwsgi_sendfile> wsgi.input = <uwsgi._Input object at 0x7ff7bcd9f850> wsgi.multiprocess = True wsgi.multithread = True wsgi.run_once = False wsgi.url_scheme = 'http' wsgi.version = '(1, 0)'

Settings: Using settings module settings ABSOLUTE_URL_OVERRIDES = {} ACCOUNT_AUTHENTICATION_METHOD = 'username_email' ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'https' ACCOUNT_EMAIL_REQUIRED = True ACCOUNT_EMAIL_VERIFICATION = 'mandatory' ACCOUNT_UNIQUE_EMAIL = True ADMINS = "(('Mailman Suite Admin', 'root@localhost'), ('bryan fields', 'bryan@bryanfields.net'))" ALLOWED_HOSTS = ['localhost', '127.0.0.1', 'nanog-mm3.keekles.org'] APPEND_SLASH = True AUTHENTICATION_BACKENDS = "('django.contrib.auth.backends.ModelBackend', 'allauth.account.auth_backends.AuthenticationBackend')" AUTH_PASSWORD_VALIDATORS = '********************' AUTH_USER_MODEL = 'auth.User' BASE_DIR = PosixPath('/opt/mailman/web') CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} CACHE_MIDDLEWARE_ALIAS = 'default' CACHE_MIDDLEWARE_KEY_PREFIX = '********************' CACHE_MIDDLEWARE_SECONDS = 600 COMPRESSORS = {'css': 'compressor.css.CssCompressor', 'js': 'compressor.js.JsCompressor'} COMPRESS_CACHEABLE_PRECOMPILERS = '()' COMPRESS_CACHE_BACKEND = 'default' COMPRESS_CACHE_KEY_FUNCTION = '********************' COMPRESS_CLEAN_CSS_ARGUMENTS = '' COMPRESS_CLEAN_CSS_BINARY = 'cleancss' COMPRESS_CLOSURE_COMPILER_ARGUMENTS = '' COMPRESS_CLOSURE_COMPILER_BINARY = 'java -jar compiler.jar' COMPRESS_CSS_HASHING_METHOD = 'mtime' COMPRESS_DATA_URI_MAX_SIZE = 1024 COMPRESS_DEBUG_TOGGLE = None COMPRESS_ENABLED = True COMPRESS_FILTERS = {'css': ['compressor.filters.css_default.CssAbsoluteFilter', 'compressor.filters.cssmin.rCSSMinFilter'], 'js': ['compressor.filters.jsmin.rJSMinFilter']} COMPRESS_JINJA2_GET_ENVIRONMENT = <function CompressorConf.JINJA2_GET_ENVIRONMENT at 0x7ff7bdb09750> COMPRESS_MINT_DELAY = 30 COMPRESS_MTIME_DELAY = 10 COMPRESS_OFFLINE = True COMPRESS_OFFLINE_CONTEXT = {'STATIC_URL': '/static/'} COMPRESS_OFFLINE_MANIFEST = 'manifest.json' COMPRESS_OFFLINE_MANIFEST_STORAGE = 'compressor.storage.OfflineManifestFileStorage' COMPRESS_OFFLINE_TIMEOUT = 31536000 COMPRESS_OUTPUT_DIR = 'CACHE' COMPRESS_PARSER = 'compressor.parser.AutoSelectParser' COMPRESS_PRECOMPILERS = "(('text/x-scss', 'sassc -t compressed {infile} {outfile}'), ('text/x-sass', 'sassc -t compressed {infile} {outfile}'))" COMPRESS_REBUILD_TIMEOUT = 2592000 COMPRESS_ROOT = '/opt/mailman/web/static' COMPRESS_STORAGE = 'compressor.storage.CompressorFileStorage' COMPRESS_TEMPLATE_FILTER_CONTEXT = {'STATIC_URL': '/static/'} COMPRESS_URL = '/static/' COMPRESS_URL_PLACEHOLDER = '/__compressor_url_placeholder__/' COMPRESS_VERBOSE = False COMPRESS_YUGLIFY_BINARY = 'yuglify' COMPRESS_YUGLIFY_CSS_ARGUMENTS = '--terminal' COMPRESS_YUGLIFY_JS_ARGUMENTS = '--terminal' COMPRESS_YUI_BINARY = 'java -jar yuicompressor.jar' COMPRESS_YUI_CSS_ARGUMENTS = '' COMPRESS_YUI_JS_ARGUMENTS = '' CSRF_COOKIE_AGE = 31449600 CSRF_COOKIE_DOMAIN = None CSRF_COOKIE_HTTPONLY = False CSRF_COOKIE_MASKED = False CSRF_COOKIE_NAME = 'csrftoken' CSRF_COOKIE_PATH = '/' CSRF_COOKIE_SAMESITE = 'Lax' CSRF_COOKIE_SECURE = False CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure' CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN' CSRF_TRUSTED_ORIGINS = ['https://nanog-mm3.keekles.org'] CSRF_USE_SESSIONS = False DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'mailmanweb', 'USER': 'mailman', 'PASSWORD': '********************', 'HOST': 'localhost', 'PORT': '5432', 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'CONN_HEALTH_CHECKS': False, 'OPTIONS': {}, 'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIGRATE': True, 'MIRROR': None, 'NAME': None}}} DATABASE_ROUTERS = [] DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440 DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000 DATA_UPLOAD_MAX_NUMBER_FILES = 100 DATETIME_FORMAT = 'N j, Y, P' DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M'] DATE_FORMAT = 'N j, Y' DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y'] DEBUG = False DEBUG_PROPAGATE_EXCEPTIONS = False DECIMAL_SEPARATOR = '.' DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' DEFAULT_CHARSET = 'utf-8' DEFAULT_EXCEPTION_REPORTER = 'django.views.debug.ExceptionReporter' DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter' DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' DEFAULT_FROM_EMAIL = 'admin@nanog-mm3.keekles.org' DEFAULT_INDEX_TABLESPACE = '' DEFAULT_TABLESPACE = '' DISALLOWED_USER_AGENTS = [] EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_HOST_PASSWORD = '********************' EMAIL_HOST_USER = '' EMAIL_PORT = 25 EMAIL_SSL_CERTFILE = None EMAIL_SSL_KEYFILE = '********************' EMAIL_SUBJECT_PREFIX = '[Django] ' EMAIL_TIMEOUT = None EMAIL_USE_LOCALTIME = False EMAIL_USE_SSL = False EMAIL_USE_TLS = False FILE_UPLOAD_DIRECTORY_PERMISSIONS = None FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler'] FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440 FILE_UPLOAD_PERMISSIONS = 420 FILE_UPLOAD_TEMP_DIR = None FILTER_VHOST = False FIRST_DAY_OF_WEEK = 0 FIXTURE_DIRS = [] FORCE_SCRIPT_NAME = None FORMAT_MODULE_PATH = None FORM_RENDERER = 'django.forms.renderers.DjangoTemplates' HAYSTACK_CONNECTIONS = {'default': {'PATH': '/opt/mailman/web/xapian_index', 'ENGINE': 'xapian_backend.XapianEngine'}} HYPERKITTY_ENABLE_GRAVATAR = True IGNORABLE_404_URLS = [] INSTALLED_APPS = ['hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.humanize', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount'] INTERNAL_IPS = [] LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ar-dz', 'Algerian Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('ckb', 'Central Kurdish (Sorani)'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('hy', 'Armenian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('ig', 'Igbo'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('ky', 'Kyrgyz'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('ms', 'Malay'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('tg', 'Tajik'), ('th', 'Thai'), ('tk', 'Turkmen'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('uz', 'Uzbek'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')] LANGUAGES_BIDI = ['he', 'ar', 'ar-dz', 'ckb', 'fa', 'ur'] LANGUAGE_CODE = 'en-us' LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_HTTPONLY = False LANGUAGE_COOKIE_NAME = 'django_language' LANGUAGE_COOKIE_PATH = '/' LANGUAGE_COOKIE_SAMESITE = None LANGUAGE_COOKIE_SECURE = False LOCALE_PATHS = [] LOGGING = {'version': 1, 'disable_existing_loggers': False, 'filters': {'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'}}, 'handlers': {'mail_admins': {'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler'}, 'file': {'level': 'INFO', 'class': 'logging.handlers.WatchedFileHandler', 'filename': '/opt/mailman/web/logs/mailmanweb.log', 'formatter': 'verbose'}, 'console': {'class': 'logging.StreamHandler', 'formatter': 'simple'}}, 'loggers': {'django.request': {'handlers': ['mail_admins', 'file'], 'level': 'ERROR', 'propagate': True}, 'django': {'handlers': ['file'], 'level': 'ERROR', 'propagate': True}, 'hyperkitty': {'handlers': ['file'], 'level': 'DEBUG', 'propagate': True}, 'postorius': {'handlers': ['console', 'file'], 'level': 'INFO'}, 'q': {'level': 'WARNING', 'propagate': False, 'handlers': ['console', 'file']}}, 'formatters': {'verbose': {'format': '%(levelname)s %(asctime)s %(process)d %(name)s %(message)s'}, 'simple': {'format': '%(levelname)s %(message)s'}}} LOGGING_CONFIG = 'logging.config.dictConfig' LOGIN_REDIRECT_URL = 'list_index' LOGIN_URL = 'account_login' LOGOUT_REDIRECT_URL = None LOGOUT_URL = 'account_logout' MAILMAN_ARCHIVER_FROM = "('127.0.0.1', '::1')" MAILMAN_ARCHIVER_KEY = '********************' MAILMAN_REST_API_PASS = '********************' MAILMAN_REST_API_URL = '********************' MAILMAN_REST_API_USER = '********************' MANAGERS = [] MEDIA_ROOT = '' MEDIA_URL = '/' MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage' MESSAGE_TAGS = {40: 'danger'} MIDDLEWARE = "('allauth.account.middleware.AccountMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')" MIGRATION_MODULES = {} MONTH_DAY_FORMAT = 'F j' NUMBER_GROUPING = 0 PASSWORD_HASHERS = '********************' PASSWORD_RESET_TIMEOUT = '********************' POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8000' PREPEND_WWW = False Q_CLUSTER = {'retry': 360, 'timeout': 300, 'save_limit': 100, 'orm': 'default', 'workers': 2} ROOT_URLCONF = 'mailman_web.urls' SECRET_KEY = '********************' SECRET_KEY_FALLBACKS = '********************' SECURE_CONTENT_TYPE_NOSNIFF = True SECURE_CROSS_ORIGIN_OPENER_POLICY = 'same-origin' SECURE_HSTS_INCLUDE_SUBDOMAINS = False SECURE_HSTS_PRELOAD = False SECURE_HSTS_SECONDS = 0 SECURE_PROXY_SSL_HEADER = None SECURE_REDIRECT_EXEMPT = [] SECURE_REFERRER_POLICY = 'same-origin' SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False SERVER_EMAIL = 'admin@nanog-mm3.keekles.org' SESSION_CACHE_ALIAS = 'default' SESSION_COOKIE_AGE = 1209600 SESSION_COOKIE_DOMAIN = None SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_NAME = 'sessionid' SESSION_COOKIE_PATH = '/' SESSION_COOKIE_SAMESITE = 'Lax' SESSION_COOKIE_SECURE = False SESSION_ENGINE = 'django.contrib.sessions.backends.db' SESSION_EXPIRE_AT_BROWSER_CLOSE = False SESSION_FILE_PATH = None SESSION_SAVE_EVERY_REQUEST = False SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer' SETTINGS_MODULE = 'settings' SHORT_DATETIME_FORMAT = 'm/d/Y P' SHORT_DATE_FORMAT = 'm/d/Y' SIGNING_BACKEND = 'django.core.signing.TimestampSigner' SILENCED_SYSTEM_CHECKS = [] SITE_ID = 1 SOCIALACCOUNT_PROVIDERS = {'openid': {'SERVERS': [{'id': 'yahoo', 'name': 'Yahoo', 'openid_url': 'http://me.yahoo.com'}]}, 'google': {'SCOPE': ['profile', 'email'], 'AUTH_PARAMS': {'access_type': 'online'}}, 'facebook': {'METHOD': 'oauth2', 'SCOPE': ['email'], 'FIELDS': ['email', 'name', 'first_name', 'last_name', 'locale', 'timezone'], 'VERSION': 'v2.4'}} STATICFILES_DIRS = '()' STATICFILES_FINDERS = "('django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'compressor.finders.CompressorFinder')" STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' STATIC_ROOT = '/opt/mailman/web/static' STATIC_URL = '/static/' STORAGES = {'default': {'BACKEND': 'django.core.files.storage.FileSystemStorage'}, 'staticfiles': {'BACKEND': 'django.contrib.staticfiles.storage.StaticFilesStorage'}} TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.media', 'django.template.context_processors.static', 'django.template.context_processors.tz', 'django.template.context_processors.csrf', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'django_mailman3.context_processors.common', 'hyperkitty.context_processors.common', 'postorius.context_processors.postorius']}}] TEST_NON_SERIALIZED_APPS = [] TEST_RUNNER = 'django.test.runner.DiscoverRunner' THOUSAND_SEPARATOR = ',' TIME_FORMAT = 'P' TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] TIME_ZONE = 'UTC' USE_DEPRECATED_PYTZ = False USE_I18N = True USE_L10N = True USE_THOUSAND_SEPARATOR = False USE_TZ = True USE_X_FORWARDED_HOST = False USE_X_FORWARDED_PORT = False WSGI_APPLICATION = 'mailman_web.wsgi.application' X_FRAME_OPTIONS = 'DENY' YEAR_MONTH_FORMAT = 'F Y'

-- Bryan Fields

727-409-1194 - Voice http://bryanfields.net

On 7/25/24 7:04 PM, Mark Sapiro wrote:

On 7/25/24 11:20, Bryan Fields wrote:

...

When this happens, I get an email error below.

This is really wierd. See below.

...

line 88, in reverse return resolver._reverse_with_prefix(view, prefix, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/resolvers.py", line 828, in _reverse_with_prefix raise NoReverseMatch(msg)

Exception Type: NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Exception Value: Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$'] Raised during: hyperkitty.views.mlist.overview

It appears to be trying to get the threads for December, 999.

What do these queries show?

SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000;

SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000;

mailman=> \c mailmanweb You are now connected to database "mailmanweb" as user "mailman". mailmanweb=> SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; id | thread_id | date_active | category_id | mailinglist_id | starting_email_id -------+----------------------------------+------------------------+-------------+----------------+------------------- 35670 | GHLCKUC7TDEBAI27SKRQN5PWX3TGB26Q | 0102-04-15 20:06:12+00 | | 3 | 150555 (1 row)

mailmanweb=> SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; id

136872 150555 (2 rows)

Does this mean it imported archives with an invalid date somehow?

Thanks,

Bryan Fields

727-409-1194 - Voice http://bryanfields.net

On 7/26/24 10:41, Bryan Fields wrote:

On 7/26/24 1:30 PM, Bryan Fields wrote:

...

mailman=> \c mailmanweb You are now connected to database "mailmanweb" as user "mailman". mailmanweb=> SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; id   |            thread_id             |      date_active       | category_id | mailinglist_id | starting_email_id -------+----------------------------------+------------------------+-------------+----------------+------------------- 35670 | GHLCKUC7TDEBAI27SKRQN5PWX3TGB26Q | 0102-04-15 20:06:12+00 |             |              3 |            150555 (1 row)

mailmanweb=> SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; id

   136872 150555 (2 rows)

Does this mean it imported archives with an invalid date somehow?

Yes, I suspect this bad date is part of the issue.

I would do a few things. First,

SELECT message_id FROM hyperkitty_email WHERE thread_id = 35670;

If this returns just the one message_id, I would then

DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;

and see if that fixes the internal server error when accessing the archive. I would also

SELECT * FROM hyperkitty_email WHERE id = 136872;

and maybe update the date in that entry.

After deleting that one message and thread, you can make a mbox with just the one message with the date corrected and run hyperkitty_import with --since 2000-01-01 and that mbox as input to add the message back to the archive.

Also, FYI, assuming those messages with very old Date: headers had reasonable unix from dates, the hyperkitty/contrib/cleanarch3 script would fix them, except you have to get the script from https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/contrib/clean... or https://www.msapiro.net/scripts/cleanarch3 because the script in the latest release has a bug.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

This thread is deliciously ironic, being about a Y2K bug emerging all these years later, and the offending message containing this:

Sat Jan 1 01:20:13 UTC 2000

GMT and CET rolled over without any major incidents. The cellular networks were busy but thats to be expected.

Regards, Neil

Spoke too soon there Neil.

I hope Neil's aware of this and able to enjoy it too.

rb

On 7/26/24 11:19 PM, Mark Sapiro wrote:

On 7/26/24 18:53, Bryan Fields wrote:

...

On 7/26/24 4:01 PM, Mark Sapiro wrote:

...

If this returns just the one message_id, I would then

DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;

It's giving an error due to a fk constraint on another table.  I'm not familiar with the db structure enough to be certain how to clean/fix it.

Yes, you need to do these first.

DELETE FROM hyperkitty_favorite WHERE thread_id = 35670; DELETE FROM hyperkitty_tagging WHERE thread_id = 35670; DELETE FROM hyperkitty_lastview WHERE thread_id = 35670; DELETE FROM hyperkitty_vote WHERE email_id = 150555;

Not all those will exist, but then you should be able to

DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;

Sill no go on it.

mailmanweb=> DELETE FROM hyperkitty_favorite WHERE thread_id = 35670; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_tagging WHERE thread_id = 35670; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_lastview WHERE thread_id = 35670; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_vote WHERE email_id = 150555; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_thread WHERE id = 35670; ERROR: update or delete on table "hyperkitty_thread" violates foreign key constraint "hyperkitty_email_thread_id_038c9aee_fk_hyperkitty_thread_id" on table "hyperkitty_email" DETAIL: Key (id)=(35670) is still referenced from table "hyperkitty_email". mailmanweb=> DELETE FROM hyperkitty_email WHERE id = 150555; ERROR: update or delete on table "hyperkitty_email" violates foreign key constraint "hyperkitty_thread_starting_email_id_fa7c55f5_fk_hyperkitt" on table "hyperkitty_thread" DETAIL: Key (id)=(150555) is still referenced from table "hyperkitty_thread".

...

I'm actually thinking I need to fix the archive mbox, then delete them and re-import them once they are cleaned.

Actually, that would be safest. Fix the mbox, delete the entire archive and re-run the import.

I'll attempt to clean this up tomorrow an clean it up/re-import.

Is there a quick/easy way to drop the entire archive from the db or must I delete the entire list?

...

...

Also, FYI, assuming those messages with very old Date: headers had reasonable unix from dates, the hyperkitty/contrib/cleanarch3 script would fix them, except you have to get the script from https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/contrib/clean... or https://www.msapiro.net/scripts/cleanarch3 because the script in the latest release has a bug.

I tried the dry run option on your script and it gave a bunch of output for bad dates.  This was absent on the one shipping with the source.  What's interesting is that pipermail seems to have no issue with this, detecting the date correctly.

If I recall correctly, pipermail checks date skew and 'fixes' the dates in the archive, but they remain off in the mbox.

I'll have to check that, but I've got at least some good leads now and tools to find suspected issues. It does seem like a bad design that importing a mailman2 mbox can cause the server to error 500.

...

Is the archive tool in pipermail's import more robust in this manner? I'd argue it's common to have missing In-Reply-To: headers where the subject and time would need to be used to infer the likely thread.  I'll agree this is a major violation of the relevant RFC's to be missing this, but many MUA's (M$) are famous for doing just this.

Yes, it's arguably a defect in HyperKitty to not implement Jamie Zawinski's threading algorithm <https://www.jwz.org/doc/threading.html>, but it doesn't.

Well that kinda blows :-(

I'm not going to take down the old archives, but I'd like at least one to be searchable, which is why we wanted to use mailman3/hyperkitty.

Thank you,

Bryan Fields

727-409-1194 - Voice http://bryanfields.net