hyperkitty threads by month question
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
I have installed a testing server for an organization I'm working with to explore migration from MM2 to MM3. Our main list has archives from 1992, and when importing the mbox into hyperkitty/mm3 I notice the archives by month only go back to July 2007, but the messages are there when I search for specific ones from the old archive in the 1992-2007 range.
Is there a better way to get this to show up, or perhaps a better format we can display old archives in that's not 400+ line drop down?
Thank you,
Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 7/9/24 11:07 AM, Bryan Fields wrote:
I have installed a testing server for an organization I'm working with to explore migration from MM2 to MM3. Our main list has archives from 1992, and when importing the mbox into hyperkitty/mm3 I notice the archives by month only go back to July 2007, but the messages are there when I search for specific ones from the old archive in the 1992-2007 range.
See this post and maybe more of that thread which may help. https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
On 7/9/24 3:39 PM, Mark Sapiro wrote:
See this post and maybe more of that thread which may help. https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
Thanks Mike, I've been looking into this, and I don't think it's that.
What I did find is that our mbox from the mailman2 list is actually a amalgamation of mailman2, and majordomo archives. The mbox is not in order, but the start of the mbox starts in 2007 (coinciding with the last displayed month). Later on in the mbox file, the archives going back to 1992 from majordomo are present.
There is no errors in the check_hk_import for this, and the messages are present in the index when I search for them and there are no errors during import.
mailman 2.1 archives works just fine with this mbox file.
Here's a link to the server in question, it's just a text box as we work through this. There's about 14k people on this list, so we want to get it right before cut over.
https://nanog-mm3.keekles.org/archives/list/mm2-mm3@nanog-mm3.keekles.org/
Any ideas?
Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
On 7/19/24 4:20 PM, Bryan Fields wrote:
Here's a link to the server in question, it's just a text box as we work through this. There's about 14k people on this list, so we want to get it right before cut over.
https://nanog-mm3.keekles.org/archives/list/mm2-mm3@nanog-mm3.keekles.org/
https://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/
Doh! gave the wrong URL. This is the correct list, and if you click "thread by month" you see it stops at July 2007.
-- Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 7/19/24 1:24 PM, Bryan Fields wrote:
On 7/19/24 4:20 PM, Bryan Fields wrote:
Here's a link to the server in question, it's just a text box as we work through this. There's about 14k people on this list, so we want to get it right before cut over.
https://nanog-mm3.keekles.org/archives/list/mm2-mm3@nanog-mm3.keekles.org/
https://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/
That URL gives me as server error. What's in the Django (mailman-web) logs for that?
Are you running all the Django periodic jobs and is qcluster running?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
Sorry for the delay, I was on some travel last week.
On 7/19/24 7:30 PM, Mark Sapiro wrote:
That URL gives me as server error. What's in the Django (mailman-web) logs for that?
I did notice hitting a 500 error on occasion, sometimes it works.
When this happens, I get an email error below.
Are you running all the Django periodic jobs and is qcluster running?
Ok, the docs said that qcluster service wasn't needed under uWSGI https://docs.mailman3.org/en/latest/install/virtualenv.html#automatically-st... qcluster does appear to be running:
218916 ? Sl 0:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 218929 ? Sl 0:00 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940851 ? Ss 0:00 /bin/sh -c /opt/mailman/venv/bin/mailman-web qcluster 3940854 ? S 0:01 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940856 ? S 117:32 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940860 ? S 0:28 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster 3940861 ? Sl 3048:11 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman-web qcluster
=== Email Error === Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /archives/list/nanog@nanog-mm3.keekles.org/
Internal Server Error: /archives/list/nanog@nanog-mm3.keekles.org/
NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$']
Request Method: GET Request URL: http://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/ Django Version: 4.2.13 Python Executable: /opt/mailman/venv/bin/uwsgi Python Version: 3.10.12 Python Path: ['.', '', '/etc/mailman3', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/opt/mailman/venv/lib/python3.10/site-packages'] Server time: Thu, 25 Jul 2024 18:16:17 +0000 Installed Applications: ['hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.humanize', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount'] Installed Middleware: ('allauth.account.middleware.AccountMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')
Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/lib/view_helpers.py", line 137, in inner return func(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/views/mlist.py", line 239, in overview return render(request, "hyperkitty/overview.html", context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/shortcuts.py", line 24, in render content = loader.render_to_string(template_name, context, request, using=using) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader.py", line 62, in render_to_string return template.render(context, request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/backends/django.py", line 61, in render return self.template.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 175, in render return self._render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader_tags.py", line 157, in render return compiled_parent._render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader_tags.py", line 63, in render result = block.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/templatetags/cache.py", line 53, in render value = self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/loader_tags.py", line 208, in render return template.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 177, in render return self._render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 1005, in <listcomp> return SafeString("".join([node.render_annotated(context) for node in self])) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/defaulttags.py", line 238, in render nodelist.append(node.render_annotated(context)) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/defaulttags.py", line 238, in render nodelist.append(node.render_annotated(context)) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) File "/opt/mailman/venv/lib/python3.10/site-packages/django/template/defaulttags.py", line 471, in render url = reverse(view_name, args=args, kwargs=kwargs, current_app=current_app) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/base.py", line 88, in reverse return resolver._reverse_with_prefix(view, prefix, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/resolvers.py", line 828, in _reverse_with_prefix raise NoReverseMatch(msg)
Exception Type: NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Exception Value: Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$'] Raised during: hyperkitty.views.mlist.overview Request information: USER: nanogadmin
GET: No GET data
POST: No POST data
FILES: No FILES data
COOKIES: csrftoken = '********************' sessionid = '********************'
META: CSRF_COOKIE = '6F7ioB7blmJwLwKQLH0DPuAoWTW8mQGI' HTTP_ACCEPT = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' HTTP_ACCEPT_ENCODING = 'gzip, deflate, br' HTTP_ACCEPT_LANGUAGE = 'en-US,en;q=0.5' HTTP_CONNECTION = 'Keep-Alive' HTTP_COOKIE = '********************' HTTP_HOST = 'nanog-mm3.keekles.org' HTTP_SEC_FETCH_DEST = 'document' HTTP_SEC_FETCH_MODE = 'navigate' HTTP_SEC_FETCH_SITE = 'none' HTTP_SEC_FETCH_USER = '?1' HTTP_UPGRADE_INSECURE_REQUESTS = '1' HTTP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:109.0) Gecko/20100101 Firefox/115.0' HTTP_X_FORWARDED_FOR = '47.206.239.202' HTTP_X_FORWARDED_HOST = 'nanog-mm3.keekles.org' HTTP_X_FORWARDED_SERVER = 'nanog-mm3.keekles.org' PATH_INFO = '/archives/list/nanog@nanog-mm3.keekles.org/' QUERY_STRING = '' REMOTE_ADDR = '127.0.0.1' REQUEST_METHOD = 'GET' REQUEST_URI = '/archives/list/nanog@nanog-mm3.keekles.org/' SCRIPT_NAME = '' SERVER_NAME = 'nanog-mm3' SERVER_PORT = '8000' SERVER_PROTOCOL = 'HTTP/1.1' uwsgi.core = 1 uwsgi.node = b'nanog-mm3' uwsgi.version = b'2.0.26' wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'> wsgi.file_wrapper = <built-in function uwsgi_sendfile> wsgi.input = <uwsgi._Input object at 0x7ff7bcd9f850> wsgi.multiprocess = True wsgi.multithread = True wsgi.run_once = False wsgi.url_scheme = 'http' wsgi.version = '(1, 0)'
Settings: Using settings module settings ABSOLUTE_URL_OVERRIDES = {} ACCOUNT_AUTHENTICATION_METHOD = 'username_email' ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'https' ACCOUNT_EMAIL_REQUIRED = True ACCOUNT_EMAIL_VERIFICATION = 'mandatory' ACCOUNT_UNIQUE_EMAIL = True ADMINS = "(('Mailman Suite Admin', 'root@localhost'), ('bryan fields', 'bryan@bryanfields.net'))" ALLOWED_HOSTS = ['localhost', '127.0.0.1', 'nanog-mm3.keekles.org'] APPEND_SLASH = True AUTHENTICATION_BACKENDS = "('django.contrib.auth.backends.ModelBackend', 'allauth.account.auth_backends.AuthenticationBackend')" AUTH_PASSWORD_VALIDATORS = '********************' AUTH_USER_MODEL = 'auth.User' BASE_DIR = PosixPath('/opt/mailman/web') CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} CACHE_MIDDLEWARE_ALIAS = 'default' CACHE_MIDDLEWARE_KEY_PREFIX = '********************' CACHE_MIDDLEWARE_SECONDS = 600 COMPRESSORS = {'css': 'compressor.css.CssCompressor', 'js': 'compressor.js.JsCompressor'} COMPRESS_CACHEABLE_PRECOMPILERS = '()' COMPRESS_CACHE_BACKEND = 'default' COMPRESS_CACHE_KEY_FUNCTION = '********************' COMPRESS_CLEAN_CSS_ARGUMENTS = '' COMPRESS_CLEAN_CSS_BINARY = 'cleancss' COMPRESS_CLOSURE_COMPILER_ARGUMENTS = '' COMPRESS_CLOSURE_COMPILER_BINARY = 'java -jar compiler.jar' COMPRESS_CSS_HASHING_METHOD = 'mtime' COMPRESS_DATA_URI_MAX_SIZE = 1024 COMPRESS_DEBUG_TOGGLE = None COMPRESS_ENABLED = True COMPRESS_FILTERS = {'css': ['compressor.filters.css_default.CssAbsoluteFilter', 'compressor.filters.cssmin.rCSSMinFilter'], 'js': ['compressor.filters.jsmin.rJSMinFilter']} COMPRESS_JINJA2_GET_ENVIRONMENT = <function CompressorConf.JINJA2_GET_ENVIRONMENT at 0x7ff7bdb09750> COMPRESS_MINT_DELAY = 30 COMPRESS_MTIME_DELAY = 10 COMPRESS_OFFLINE = True COMPRESS_OFFLINE_CONTEXT = {'STATIC_URL': '/static/'} COMPRESS_OFFLINE_MANIFEST = 'manifest.json' COMPRESS_OFFLINE_MANIFEST_STORAGE = 'compressor.storage.OfflineManifestFileStorage' COMPRESS_OFFLINE_TIMEOUT = 31536000 COMPRESS_OUTPUT_DIR = 'CACHE' COMPRESS_PARSER = 'compressor.parser.AutoSelectParser' COMPRESS_PRECOMPILERS = "(('text/x-scss', 'sassc -t compressed {infile} {outfile}'), ('text/x-sass', 'sassc -t compressed {infile} {outfile}'))" COMPRESS_REBUILD_TIMEOUT = 2592000 COMPRESS_ROOT = '/opt/mailman/web/static' COMPRESS_STORAGE = 'compressor.storage.CompressorFileStorage' COMPRESS_TEMPLATE_FILTER_CONTEXT = {'STATIC_URL': '/static/'} COMPRESS_URL = '/static/' COMPRESS_URL_PLACEHOLDER = '/__compressor_url_placeholder__/' COMPRESS_VERBOSE = False COMPRESS_YUGLIFY_BINARY = 'yuglify' COMPRESS_YUGLIFY_CSS_ARGUMENTS = '--terminal' COMPRESS_YUGLIFY_JS_ARGUMENTS = '--terminal' COMPRESS_YUI_BINARY = 'java -jar yuicompressor.jar' COMPRESS_YUI_CSS_ARGUMENTS = '' COMPRESS_YUI_JS_ARGUMENTS = '' CSRF_COOKIE_AGE = 31449600 CSRF_COOKIE_DOMAIN = None CSRF_COOKIE_HTTPONLY = False CSRF_COOKIE_MASKED = False CSRF_COOKIE_NAME = 'csrftoken' CSRF_COOKIE_PATH = '/' CSRF_COOKIE_SAMESITE = 'Lax' CSRF_COOKIE_SECURE = False CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure' CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN' CSRF_TRUSTED_ORIGINS = ['https://nanog-mm3.keekles.org'] CSRF_USE_SESSIONS = False DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'mailmanweb', 'USER': 'mailman', 'PASSWORD': '********************', 'HOST': 'localhost', 'PORT': '5432', 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'CONN_HEALTH_CHECKS': False, 'OPTIONS': {}, 'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIGRATE': True, 'MIRROR': None, 'NAME': None}}} DATABASE_ROUTERS = [] DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440 DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000 DATA_UPLOAD_MAX_NUMBER_FILES = 100 DATETIME_FORMAT = 'N j, Y, P' DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M'] DATE_FORMAT = 'N j, Y' DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y'] DEBUG = False DEBUG_PROPAGATE_EXCEPTIONS = False DECIMAL_SEPARATOR = '.' DEFAULT_AUTO_FIELD = 'django.db.models.AutoField' DEFAULT_CHARSET = 'utf-8' DEFAULT_EXCEPTION_REPORTER = 'django.views.debug.ExceptionReporter' DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter' DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' DEFAULT_FROM_EMAIL = 'admin@nanog-mm3.keekles.org' DEFAULT_INDEX_TABLESPACE = '' DEFAULT_TABLESPACE = '' DISALLOWED_USER_AGENTS = [] EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_HOST_PASSWORD = '********************' EMAIL_HOST_USER = '' EMAIL_PORT = 25 EMAIL_SSL_CERTFILE = None EMAIL_SSL_KEYFILE = '********************' EMAIL_SUBJECT_PREFIX = '[Django] ' EMAIL_TIMEOUT = None EMAIL_USE_LOCALTIME = False EMAIL_USE_SSL = False EMAIL_USE_TLS = False FILE_UPLOAD_DIRECTORY_PERMISSIONS = None FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler'] FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440 FILE_UPLOAD_PERMISSIONS = 420 FILE_UPLOAD_TEMP_DIR = None FILTER_VHOST = False FIRST_DAY_OF_WEEK = 0 FIXTURE_DIRS = [] FORCE_SCRIPT_NAME = None FORMAT_MODULE_PATH = None FORM_RENDERER = 'django.forms.renderers.DjangoTemplates' HAYSTACK_CONNECTIONS = {'default': {'PATH': '/opt/mailman/web/xapian_index', 'ENGINE': 'xapian_backend.XapianEngine'}} HYPERKITTY_ENABLE_GRAVATAR = True IGNORABLE_404_URLS = [] INSTALLED_APPS = ['hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.humanize', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount'] INTERNAL_IPS = [] LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ar-dz', 'Algerian Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('ckb', 'Central Kurdish (Sorani)'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('hy', 'Armenian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('ig', 'Igbo'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('ky', 'Kyrgyz'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('ms', 'Malay'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('tg', 'Tajik'), ('th', 'Thai'), ('tk', 'Turkmen'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('uz', 'Uzbek'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')] LANGUAGES_BIDI = ['he', 'ar', 'ar-dz', 'ckb', 'fa', 'ur'] LANGUAGE_CODE = 'en-us' LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_HTTPONLY = False LANGUAGE_COOKIE_NAME = 'django_language' LANGUAGE_COOKIE_PATH = '/' LANGUAGE_COOKIE_SAMESITE = None LANGUAGE_COOKIE_SECURE = False LOCALE_PATHS = [] LOGGING = {'version': 1, 'disable_existing_loggers': False, 'filters': {'require_debug_false': {'()': 'django.utils.log.RequireDebugFalse'}}, 'handlers': {'mail_admins': {'level': 'ERROR', 'filters': ['require_debug_false'], 'class': 'django.utils.log.AdminEmailHandler'}, 'file': {'level': 'INFO', 'class': 'logging.handlers.WatchedFileHandler', 'filename': '/opt/mailman/web/logs/mailmanweb.log', 'formatter': 'verbose'}, 'console': {'class': 'logging.StreamHandler', 'formatter': 'simple'}}, 'loggers': {'django.request': {'handlers': ['mail_admins', 'file'], 'level': 'ERROR', 'propagate': True}, 'django': {'handlers': ['file'], 'level': 'ERROR', 'propagate': True}, 'hyperkitty': {'handlers': ['file'], 'level': 'DEBUG', 'propagate': True}, 'postorius': {'handlers': ['console', 'file'], 'level': 'INFO'}, 'q': {'level': 'WARNING', 'propagate': False, 'handlers': ['console', 'file']}}, 'formatters': {'verbose': {'format': '%(levelname)s %(asctime)s %(process)d %(name)s %(message)s'}, 'simple': {'format': '%(levelname)s %(message)s'}}} LOGGING_CONFIG = 'logging.config.dictConfig' LOGIN_REDIRECT_URL = 'list_index' LOGIN_URL = 'account_login' LOGOUT_REDIRECT_URL = None LOGOUT_URL = 'account_logout' MAILMAN_ARCHIVER_FROM = "('127.0.0.1', '::1')" MAILMAN_ARCHIVER_KEY = '********************' MAILMAN_REST_API_PASS = '********************' MAILMAN_REST_API_URL = '********************' MAILMAN_REST_API_USER = '********************' MANAGERS = [] MEDIA_ROOT = '' MEDIA_URL = '/' MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage' MESSAGE_TAGS = {40: 'danger'} MIDDLEWARE = "('allauth.account.middleware.AccountMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')" MIGRATION_MODULES = {} MONTH_DAY_FORMAT = 'F j' NUMBER_GROUPING = 0 PASSWORD_HASHERS = '********************' PASSWORD_RESET_TIMEOUT = '********************' POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8000' PREPEND_WWW = False Q_CLUSTER = {'retry': 360, 'timeout': 300, 'save_limit': 100, 'orm': 'default', 'workers': 2} ROOT_URLCONF = 'mailman_web.urls' SECRET_KEY = '********************' SECRET_KEY_FALLBACKS = '********************' SECURE_CONTENT_TYPE_NOSNIFF = True SECURE_CROSS_ORIGIN_OPENER_POLICY = 'same-origin' SECURE_HSTS_INCLUDE_SUBDOMAINS = False SECURE_HSTS_PRELOAD = False SECURE_HSTS_SECONDS = 0 SECURE_PROXY_SSL_HEADER = None SECURE_REDIRECT_EXEMPT = [] SECURE_REFERRER_POLICY = 'same-origin' SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False SERVER_EMAIL = 'admin@nanog-mm3.keekles.org' SESSION_CACHE_ALIAS = 'default' SESSION_COOKIE_AGE = 1209600 SESSION_COOKIE_DOMAIN = None SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_NAME = 'sessionid' SESSION_COOKIE_PATH = '/' SESSION_COOKIE_SAMESITE = 'Lax' SESSION_COOKIE_SECURE = False SESSION_ENGINE = 'django.contrib.sessions.backends.db' SESSION_EXPIRE_AT_BROWSER_CLOSE = False SESSION_FILE_PATH = None SESSION_SAVE_EVERY_REQUEST = False SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer' SETTINGS_MODULE = 'settings' SHORT_DATETIME_FORMAT = 'm/d/Y P' SHORT_DATE_FORMAT = 'm/d/Y' SIGNING_BACKEND = 'django.core.signing.TimestampSigner' SILENCED_SYSTEM_CHECKS = [] SITE_ID = 1 SOCIALACCOUNT_PROVIDERS = {'openid': {'SERVERS': [{'id': 'yahoo', 'name': 'Yahoo', 'openid_url': 'http://me.yahoo.com'}]}, 'google': {'SCOPE': ['profile', 'email'], 'AUTH_PARAMS': {'access_type': 'online'}}, 'facebook': {'METHOD': 'oauth2', 'SCOPE': ['email'], 'FIELDS': ['email', 'name', 'first_name', 'last_name', 'locale', 'timezone'], 'VERSION': 'v2.4'}} STATICFILES_DIRS = '()' STATICFILES_FINDERS = "('django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'compressor.finders.CompressorFinder')" STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.StaticFilesStorage' STATIC_ROOT = '/opt/mailman/web/static' STATIC_URL = '/static/' STORAGES = {'default': {'BACKEND': 'django.core.files.storage.FileSystemStorage'}, 'staticfiles': {'BACKEND': 'django.contrib.staticfiles.storage.StaticFilesStorage'}} TEMPLATES = [{'BACKEND': 'django.template.backends.django.DjangoTemplates', 'DIRS': [], 'APP_DIRS': True, 'OPTIONS': {'context_processors': ['django.template.context_processors.debug', 'django.template.context_processors.i18n', 'django.template.context_processors.media', 'django.template.context_processors.static', 'django.template.context_processors.tz', 'django.template.context_processors.csrf', 'django.template.context_processors.request', 'django.contrib.auth.context_processors.auth', 'django.contrib.messages.context_processors.messages', 'django_mailman3.context_processors.common', 'hyperkitty.context_processors.common', 'postorius.context_processors.postorius']}}] TEST_NON_SERIALIZED_APPS = [] TEST_RUNNER = 'django.test.runner.DiscoverRunner' THOUSAND_SEPARATOR = ',' TIME_FORMAT = 'P' TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] TIME_ZONE = 'UTC' USE_DEPRECATED_PYTZ = False USE_I18N = True USE_L10N = True USE_THOUSAND_SEPARATOR = False USE_TZ = True USE_X_FORWARDED_HOST = False USE_X_FORWARDED_PORT = False WSGI_APPLICATION = 'mailman_web.wsgi.application' X_FRAME_OPTIONS = 'DENY' YEAR_MONTH_FORMAT = 'F Y'
-- Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 7/25/24 11:20, Bryan Fields wrote:
Sorry for the delay, I was on some travel last week.
On 7/19/24 7:30 PM, Mark Sapiro wrote:
That URL gives me as server error. What's in the Django (mailman-web) logs for that?
I did notice hitting a 500 error on occasion, sometimes it works.
I just tried it twice more and got the error both times.
When this happens, I get an email error below.
This is really wierd. See below.
Are you running all the Django periodic jobs and is qcluster running?
Ok, the docs said that qcluster service wasn't needed under uWSGI https://docs.mailman3.org/en/latest/install/virtualenv.html#automatically-st... qcluster does appear to be running:
Right, you don't need to create a separate service for it with our recommended uWSGI configuration because uWSGI runs it, but it does need to run.
...
line 88, in reverse return resolver._reverse_with_prefix(view, prefix, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/resolvers.py", line 828, in _reverse_with_prefix raise NoReverseMatch(msg)
Exception Type: NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Exception Value: Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$'] Raised during: hyperkitty.views.mlist.overview
It appears to be trying to get the threads for December, 999.
What do these queries show?
SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000;
SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000;
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
On 7/25/24 7:04 PM, Mark Sapiro wrote:
On 7/25/24 11:20, Bryan Fields wrote:
When this happens, I get an email error below.
This is really wierd. See below.
line 88, in reverse return resolver._reverse_with_prefix(view, prefix, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/django/urls/resolvers.py", line 828, in _reverse_with_prefix raise NoReverseMatch(msg)
Exception Type: NoReverseMatch at /archives/list/nanog@nanog-mm3.keekles.org/ Exception Value: Reverse for 'hk_archives_with_month' with keyword arguments '{'year': 999, 'mlist_fqdn': 'nanog@nanog-mm3.keekles.org', 'month': 12}' not found. 2 pattern(s) tried: ['archives/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$', 'hyperkitty/list/(?P<mlist_fqdn>[^/@]+@[^/@]+)/(?P<year>\\d{4})/(?P<month>\\d\\d?)/$'] Raised during: hyperkitty.views.mlist.overview
It appears to be trying to get the threads for December, 999.
What do these queries show?
SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000;
SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000;
mailman=> \c mailmanweb You are now connected to database "mailmanweb" as user "mailman". mailmanweb=> SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; id | thread_id | date_active | category_id | mailinglist_id | starting_email_id -------+----------------------------------+------------------------+-------------+----------------+------------------- 35670 | GHLCKUC7TDEBAI27SKRQN5PWX3TGB26Q | 0102-04-15 20:06:12+00 | | 3 | 150555 (1 row)
mailmanweb=> SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; id
136872 150555 (2 rows)
Does this mean it imported archives with an invalid date somehow?
Thanks,
Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
On 7/26/24 1:30 PM, Bryan Fields wrote:
mailman=> \c mailmanweb You are now connected to database "mailmanweb" as user "mailman". mailmanweb=> SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; id | thread_id | date_active | category_id | mailinglist_id | starting_email_id -------+----------------------------------+------------------------+-------------+----------------+------------------- 35670 | GHLCKUC7TDEBAI27SKRQN5PWX3TGB26Q | 0102-04-15 20:06:12+00 | | 3 | 150555 (1 row)
mailmanweb=> SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; id
136872 150555 (2 rows)
Does this mean it imported archives with an invalid date somehow?
https://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/thre...
I had a look at the original MBOX import of this archive is "Date: Mon, 15 Apr 102 15:06:12 CDT". It should likely be 2002 based on the rest of the headers
-- Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 7/26/24 10:41, Bryan Fields wrote:
On 7/26/24 1:30 PM, Bryan Fields wrote:
mailman=> \c mailmanweb You are now connected to database "mailmanweb" as user "mailman". mailmanweb=> SELECT * FROM hyperkitty_thread WHERE EXTRACT(year FROM date_active) < 1000; id | thread_id | date_active | category_id | mailinglist_id | starting_email_id -------+----------------------------------+------------------------+-------------+----------------+------------------- 35670 | GHLCKUC7TDEBAI27SKRQN5PWX3TGB26Q | 0102-04-15 20:06:12+00 | | 3 | 150555 (1 row)
mailmanweb=> SELECT id FROM hyperkitty_email WHERE EXTRACT(year FROM date) < 1000; id
136872 150555 (2 rows)
Does this mean it imported archives with an invalid date somehow?
Yes, I suspect this bad date is part of the issue.
I would do a few things. First,
SELECT message_id FROM hyperkitty_email WHERE thread_id = 35670;
If this returns just the one message_id, I would then
DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;
and see if that fixes the internal server error when accessing the archive. I would also
SELECT * FROM hyperkitty_email WHERE id = 136872;
and maybe update the date
in that entry.
After deleting that one message and thread, you can make a mbox with
just the one message with the date corrected and run hyperkitty_import
with --since 2000-01-01
and that mbox as input to add the message back
to the archive.
Also, FYI, assuming those messages with very old Date: headers had reasonable unix from dates, the hyperkitty/contrib/cleanarch3 script would fix them, except you have to get the script from https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/contrib/clean... or https://www.msapiro.net/scripts/cleanarch3 because the script in the latest release has a bug.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
On 7/26/24 4:01 PM, Mark Sapiro wrote:
Yes, I suspect this bad date is part of the issue.
I would do a few things. First,
SELECT message_id FROM hyperkitty_email WHERE thread_id = 35670;
mailmanweb=> SELECT message_id FROM hyperkitty_email WHERE thread_id = 35670; message_id
200204151506953.SM01412@sunchar.com
Yep, that matches the bad date message ID.
If this returns just the one message_id, I would then
DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;
It's giving an error due to a fk constraint on another table. I'm not familiar with the db structure enough to be certain how to clean/fix it.
mailmanweb=> DELETE FROM hyperkitty_thread WHERE id = 35670; ERROR: update or delete on table "hyperkitty_thread" violates foreign key constraint "hyperkitty_lastview_thread_id_5bd4f0ad_fk_hyperkitty_thread_id" on table "hyperkitty_lastview" DETAIL: Key (id)=(35670) is still referenced from table "hyperkitty_lastview". mailmanweb=> DELETE FROM hyperkitty_email WHERE id = 150555; ERROR: update or delete on table "hyperkitty_email" violates foreign key constraint "hyperkitty_thread_starting_email_id_fa7c55f5_fk_hyperkitt" on table "hyperkitty_thread" DETAIL: Key (id)=(150555) is still referenced from table "hyperkitty_thread".
and see if that fixes the internal server error when accessing the archive. I would also
SELECT * FROM hyperkitty_email WHERE id = 136872;
mailmanweb=> SELECT * FROM hyperkitty_email WHERE id = 136872; id | message_id | message_id_hash | subject | content | date | timezone | in_reply_to | archived_date | thread_depth | thread_order | mailinglist_id | parent_id | sender_id | thread_id | sender_name --------+------------------------------------------+----------------------------------+------------------+-------------------------------------------------------------------+------------------------+----------+-------------+------------------------+--------------+--------------+----------------+-----------+-------------------------+-----------+--------------- 136872 | 200001010102.BAA05112@genesis.domino.org | SGZJVBCSRYSHGHR4BB2632INFGYWOJYH | UK GMT roll over | +| 0100-01-01 01:02:34+00 | 0 | | 1999-12-31 20:20:13+00 | 0 | 0 | 3 | | neil@genesis.domino.org | 31067 | Neil J. McRae | | | | GMT and CET rolled over without any major incidents. The cellular+| | | | | | | | | | | | | | | networks were busy but thats to be expected. +| | | | | | | | | | | | | | | +| | | | | | | | | | | | | | | Regards, +| | | | | | | | | | | | | | | Neil. +| | | | | | | | | | | | | | | -- +| | | | | | | | | | | | | | | Neil J. McRae - Alive and Kicking. +| | | | | | | | | | | | | | | neil@DOMINO.ORG +| | | | | | | | | | | | | | | +| | | | | | | | | | | | | | | | | | | | | | | | | |
That message had "Date: Sat, 1 Jan 100 01:02:34 +0000 (GMT)" in the mbox. It looks like it should be year 2000 from the rest of the headers
This message was found by your script too: Date: changed at line 13785764 Date: Sat, 1 Jan 100 01:02:34 +0000 (GMT) Date: Fri, 31 Dec 1999 20:20:13 -0000
and maybe update the
date
in that entry.
I'm actually thinking I need to fix the archive mbox, then delete them and re-import them once they are cleaned.
Also, FYI, assuming those messages with very old Date: headers had reasonable unix from dates, the hyperkitty/contrib/cleanarch3 script would fix them, except you have to get the script from https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/contrib/clean... or https://www.msapiro.net/scripts/cleanarch3 because the script in the latest release has a bug.
I tried the dry run option on your script and it gave a bunch of output for bad dates. This was absent on the one shipping with the source. What's interesting is that pipermail seems to have no issue with this, detecting the date correctly. https://mailman.nanog.org/pipermail/nanog/2000-January/137630.html
One other thing, I did notice a number of archives not showing up in sequence due to missing "In-Reply-To:" headers from the source. The first message in this is one like this, and pipermail in mmm2 seems to handle this by referencing the subject, https://mailman.nanog.org/pipermail/nanog/2002-April/151325.html, where as in hyperkitty it's an orphaned thread.
Is the archive tool in pipermail's import more robust in this manner? I'd argue it's common to have missing In-Reply-To: headers where the subject and time would need to be used to infer the likely thread. I'll agree this is a major violation of the relevant RFC's to be missing this, but many MUA's (M$) are famous for doing just this.
Thank you,
Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/5ae2fa9e6ffb9118acfaafcebad6edbf.jpg?s=120&d=mm&r=g)
This thread is deliciously ironic, being about a Y2K bug emerging all these years later, and the offending message containing this:
Sat Jan 1 01:20:13 UTC 2000
GMT and CET rolled over without any major incidents. The cellular networks were busy but thats to be expected.
Regards, Neil
Spoke too soon there Neil.
I hope Neil's aware of this and able to enjoy it too.
rb
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 7/26/24 18:53, Bryan Fields wrote:
On 7/26/24 4:01 PM, Mark Sapiro wrote:
If this returns just the one message_id, I would then
DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;
It's giving an error due to a fk constraint on another table. I'm not familiar with the db structure enough to be certain how to clean/fix it.
Yes, you need to do these first.
DELETE FROM hyperkitty_favorite WHERE thread_id = 35670; DELETE FROM hyperkitty_tagging WHERE thread_id = 35670; DELETE FROM hyperkitty_lastview WHERE thread_id = 35670; DELETE FROM hyperkitty_vote WHERE email_id = 150555;
Not all those will exist, but then you should be able to
DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;
I'm actually thinking I need to fix the archive mbox, then delete them and re-import them once they are cleaned.
Actually, that would be safest. Fix the mbox, delete the entire archive and re-run the import.
Also, FYI, assuming those messages with very old Date: headers had reasonable unix from dates, the hyperkitty/contrib/cleanarch3 script would fix them, except you have to get the script from https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/contrib/clean... or https://www.msapiro.net/scripts/cleanarch3 because the script in the latest release has a bug.
I tried the dry run option on your script and it gave a bunch of output for bad dates. This was absent on the one shipping with the source. What's interesting is that pipermail seems to have no issue with this, detecting the date correctly.
If I recall correctly, pipermail checks date skew and 'fixes' the dates in the archive, but they remain off in the mbox.
https://mailman.nanog.org/pipermail/nanog/2000-January/137630.html One other thing, I did notice a number of archives not showing up in sequence due to missing "In-Reply-To:" headers from the source. The first message in this is one like this, and pipermail in mmm2 seems to handle this by referencing the subject, https://mailman.nanog.org/pipermail/nanog/2002-April/151325.html, where as in hyperkitty it's an orphaned thread.
Yes, HyperKitty does not do threading by subject matching. It only threads based on In-Reply-To: and absent In-Reply-To:, only the last entry in References: if any.
Is the archive tool in pipermail's import more robust in this manner?
I'd argue it's common to have missing In-Reply-To: headers where the subject and time would need to be used to infer the likely thread. I'll agree this is a major violation of the relevant RFC's to be missing this, but many MUA's (M$) are famous for doing just this.
Yes, it's arguably a defect in HyperKitty to not implement Jamie Zawinski's threading algorithm <https://www.jwz.org/doc/threading.html>, but it doesn't.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
![](https://secure.gravatar.com/avatar/c4af5e29ad2bf1ea52b555562b4b59f4.jpg?s=120&d=mm&r=g)
On 7/26/24 11:19 PM, Mark Sapiro wrote:
On 7/26/24 18:53, Bryan Fields wrote:
On 7/26/24 4:01 PM, Mark Sapiro wrote:
If this returns just the one message_id, I would then
DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;
It's giving an error due to a fk constraint on another table. I'm not familiar with the db structure enough to be certain how to clean/fix it.
Yes, you need to do these first.
DELETE FROM hyperkitty_favorite WHERE thread_id = 35670; DELETE FROM hyperkitty_tagging WHERE thread_id = 35670; DELETE FROM hyperkitty_lastview WHERE thread_id = 35670; DELETE FROM hyperkitty_vote WHERE email_id = 150555;
Not all those will exist, but then you should be able to
DELETE FROM hyperkitty_thread WHERE id = 35670; DELETE FROM hyperkitty_email WHERE id = 150555;
Sill no go on it.
mailmanweb=> DELETE FROM hyperkitty_favorite WHERE thread_id = 35670; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_tagging WHERE thread_id = 35670; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_lastview WHERE thread_id = 35670; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_vote WHERE email_id = 150555; DELETE 0 mailmanweb=> DELETE FROM hyperkitty_thread WHERE id = 35670; ERROR: update or delete on table "hyperkitty_thread" violates foreign key constraint "hyperkitty_email_thread_id_038c9aee_fk_hyperkitty_thread_id" on table "hyperkitty_email" DETAIL: Key (id)=(35670) is still referenced from table "hyperkitty_email". mailmanweb=> DELETE FROM hyperkitty_email WHERE id = 150555; ERROR: update or delete on table "hyperkitty_email" violates foreign key constraint "hyperkitty_thread_starting_email_id_fa7c55f5_fk_hyperkitt" on table "hyperkitty_thread" DETAIL: Key (id)=(150555) is still referenced from table "hyperkitty_thread".
I'm actually thinking I need to fix the archive mbox, then delete them and re-import them once they are cleaned.
Actually, that would be safest. Fix the mbox, delete the entire archive and re-run the import.
I'll attempt to clean this up tomorrow an clean it up/re-import.
Is there a quick/easy way to drop the entire archive from the db or must I delete the entire list?
Also, FYI, assuming those messages with very old Date: headers had reasonable unix from dates, the hyperkitty/contrib/cleanarch3 script would fix them, except you have to get the script from https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/contrib/clean... or https://www.msapiro.net/scripts/cleanarch3 because the script in the latest release has a bug.
I tried the dry run option on your script and it gave a bunch of output for bad dates. This was absent on the one shipping with the source. What's interesting is that pipermail seems to have no issue with this, detecting the date correctly.
If I recall correctly, pipermail checks date skew and 'fixes' the dates in the archive, but they remain off in the mbox.
I'll have to check that, but I've got at least some good leads now and tools to find suspected issues. It does seem like a bad design that importing a mailman2 mbox can cause the server to error 500.
Is the archive tool in pipermail's import more robust in this manner? I'd argue it's common to have missing In-Reply-To: headers where the subject and time would need to be used to infer the likely thread. I'll agree this is a major violation of the relevant RFC's to be missing this, but many MUA's (M$) are famous for doing just this.
Yes, it's arguably a defect in HyperKitty to not implement Jamie Zawinski's threading algorithm <https://www.jwz.org/doc/threading.html>, but it doesn't.
Well that kinda blows :-(
I'm not going to take down the old archives, but I'd like at least one to be searchable, which is why we wanted to use mailman3/hyperkitty.
Thank you,
Bryan Fields
727-409-1194 - Voice http://bryanfields.net
![](https://secure.gravatar.com/avatar/56f108518d7ee2544412cc80978e3182.jpg?s=120&d=mm&r=g)
On 7/26/24 20:53, Bryan Fields wrote:
mailmanweb=> DELETE FROM hyperkitty_thread WHERE id = 35670; ERROR: update or delete on table "hyperkitty_thread" violates foreign key constraint "hyperkitty_email_thread_id_038c9aee_fk_hyperkitty_thread_id" on table "hyperkitty_email" DETAIL: Key (id)=(35670) is still referenced from table "hyperkitty_email". mailmanweb=> DELETE FROM hyperkitty_email WHERE id = 150555; ERROR: update or delete on table "hyperkitty_email" violates foreign key constraint "hyperkitty_thread_starting_email_id_fa7c55f5_fk_hyperkitt" on table "hyperkitty_thread" DETAIL: Key (id)=(150555) is still referenced from table "hyperkitty_thread".
Well, that's a catch 22, you can't delete the email because it's referenced by the thread and you can't delete the thread because it's referenced by the email. You could always update those entries to remove the references, but deleting and reimporting is probably better.
I'll attempt to clean this up tomorrow an clean it up/re-import.
Is there a quick/easy way to drop the entire archive from the db or must I delete the entire list?
If you can get to https://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/ as a Django staff user or superuser, there is a Delete Archive button to delete the entire archive.
Even if you can't get there, you may be able to get to the confirmation page at https://nanog-mm3.keekles.org/archives/list/nanog@nanog-mm3.keekles.org/dele...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Bryan Fields
-
Mark Sapiro
-
Ron / BCLUG