Exception when rendering certain emails ([Django] ERROR (EXTERNAL IP): Internal Server Error...)
Hi,
I get exceptions logged when certain pages in the email archives (imported from Mailman 2) are opened. For example, this thread: https://lists.tdwg.org/archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJP...
Could this be a bug in parsing the message?
It's deployed using the maxking Docker packages, v0.5.3b1.
Thanks,
Matt
Log follows:
Internal Server Error: /archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJPACPZZN7UDNKJIPIFNTM7CA63/replies
AssertionError at /archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJPACPZZN7UDNKJIPIFNTM7CA63/replies No exception message supplied
Request Method: GET Request URL: https://lists.tdwg.org/archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJPACPZZN7UDNKJIPIFNTM7CA63/replies?sort=thread&last_view= Django Version: 4.2.20 Python Executable: /usr/bin/uwsgi Python Version: 3.12.10 Python Path: ['.', '', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/usr/lib/python3.12/site-packages'] Server time: Wed, 30 Apr 2025 15:04:09 +0000 Installed Applications: ['hyperkitty', 'postorius', 'django_mailman3', 'django.contrib.admin', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.messages', 'django.contrib.staticfiles', 'django.contrib.humanize', 'rest_framework', 'django_gravatar', 'compressor', 'haystack', 'django_extensions', 'django_q', 'allauth', 'allauth.account', 'allauth.socialaccount', 'django_mailman3.lib.auth.fedora', 'allauth.socialaccount.providers.openid', 'allauth.socialaccount.providers.github', 'allauth.socialaccount.providers.gitlab', 'allauth.socialaccount.providers.google'] Installed Middleware: ('django.contrib.sessions.middleware.SessionMiddleware', 'django.middleware.common.CommonMiddleware', 'django.middleware.csrf.CsrfViewMiddleware', 'django.middleware.locale.LocaleMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.contrib.messages.middleware.MessageMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'django.middleware.security.SecurityMiddleware', 'allauth.account.middleware.AccountMiddleware', 'django_mailman3.middleware.TimezoneMiddleware', 'postorius.middleware.PostoriusMiddleware')
Traceback (most recent call last): File "/usr/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/hyperkitty/lib/view_helpers.py", line 137, in inner return func(request, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/hyperkitty/views/thread.py", line 269, in replies replies_html = replies_tpl.render(context, request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/backends/django.py", line 61, in render return self.template.render(context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 175, in render return self._render(context) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/defaulttags.py", line 238, in render nodelist.append(node.render_annotated(context)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/loader_tags.py", line 208, in render return template.render(context) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 177, in render return self._render(context) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 167, in _render return self.nodelist.render(context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 1005, in render return SafeString("".join([node.render_annotated(context) for node in self])) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 966, in render_annotated return self.render(context) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 1064, in render output = self.filter_expression.resolve(context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/django/template/base.py", line 742, in resolve new_obj = func(obj, *arg_vals) ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/hyperkitty/templatetags/decorate.py", line 42, in render return mark_safe(text_renderer(content)) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/markdown.py", line 120, in __call__ return self.parse(s)[0] ^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/markdown.py", line 93, in parse self.block.parse(state) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/block_parser.py", line 457, in parse end_pos2 = self.parse_method(m, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/core.py", line 216, in parse_method return func(m, state) ^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/block_parser.py", line 368, in parse_block_quote text, end_pos = self.extract_block_quote(m, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/block_parser.py", line 345, in extract_block_quote end_pos = self.parse_method(m4, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/core.py", line 216, in parse_method return func(m, state) ^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/block_parser.py", line 387, in parse_list return parse_list(self, m, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/list_parser.py", line 64, in parse_list groups = _parse_list_item(block, bullet, groups, token, state, rules) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/list_parser.py", line 174, in _parse_list_item block.parse(child, rules) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/block_parser.py", line 457, in parse end_pos2 = self.parse_method(m, state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/site-packages/mistune/core.py", line 214, in parse_method assert lastgroup ^^^^^^^^^
Exception Type: AssertionError at /archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJPACPZZN7UDNKJIPIFNTM7CA63/replies
Raised during: hyperkitty.views.thread.replies Request information: USER: mailman-admin
GET: sort = 'thread' last_view = ''
POST: No POST data
FILES: No FILES data
COOKIES: csrftoken = '********************' sessionid = '********************'
Matt Blissett via Mailman-users writes:
I get exceptions logged when certain pages in the email archives (imported from Mailman 2) are opened. For example, this thread:
https://lists.tdwg.org/archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJP...
Could this be a bug in parsing the message?
It does not appear to be a bug in parsing this message. Rather, it's trying to build the reply tree and raising the exception then.
This:
File "/usr/lib/python3.12/site-packages/mistune/markdown.py", line 120, in __call__ return self.parse(s)[0]
suggests it's trying to parse Markdown out of a non-markdown file. This:
File "/usr/lib/python3.12/site-packages/mistune/core.py", line 214, in parse_method assert lastgroup ^^^^^^^^^
Exception Type: AssertionError at /archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJPACPZZN7UDNKJIPIFNTM7CA63/replies
doesn't make sense to me, as core.py has only 207 lines in my installation, and there is no assert in the parse_method function. Check what version you have.
A very few versions of mistune have caused problems in the past. I wonder if there's a version incompatibility here. My installation has mistune v3.0.2. (Since the exception triggers only on certain files, you should not consider that version "known good".) It's also possible that it's an issue in HyperKitty, as HyperKitty does tweak the mistune parser with its own rules to handle email-specific syntax such as .signature blocks and multipart/signed.
If you do not need the Markdown rendering, you can disable it (per list). See https://docs.mailman3.org/projects/hyperkitty/en/latest/rendering.html for how and for more information on HyperKitty's markdown capabilities.
If that's not helpful, Mark or Abhilash will likely be along soon with more expert assistance.
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan
Stephen J. Turnbull wrote:
Matt Blissett via Mailman-users writes:
I get exceptions logged when certain pages in the email archives ... This: File "/usr/lib/python3.12/site-packages/mistune/markdown.py", line 120, in __call__ return self.parse(s)[0] suggests it's trying to parse Markdown out of a non-markdown file. This: File "/usr/lib/python3.12/site-packages/mistune/core.py", line 214, in parse_method assert lastgroup ^^^^^^^^^ Exception Type: AssertionError at /archives/list/tdwg-tag@lists.tdwg.org/thread/HZCJQBJPACPZZN7UDNKJIPIFNTM7CA63/replies doesn't make sense to me, as core.py has only 207 lines in my installation, and there is no assert in the parse_method function. Check what version you have.
Thanks — it's version 3.1.3, so it's these lines: https://github.com/lepture/mistune/blob/v3.1.3/src/mistune/core.py#L212-L216
If you do not need the Markdown rendering, you can disable it (per list).
"Archive Rendering Mode" for this list is already set to "Plain text".
-- Matt
Matt Blissett via Mailman-users writes:
Thanks ? it's version 3.1.3, so it's these lines: https://github.com/lepture/mistune/blob/v3.1.3/src/mistune/core.py#L212-L216
OK. The version bump and the assert suggest to me that they've made significant changes. I don't know much about docker, but if you can arrange for the container to be built with "mistune<3.1" in the requirements.txt file or other file specifying the Python packages, that might work better as I cannot recall reports of this kind of problem. (No promises, just going on the "significant changes" guess.)
If you do not need the Markdown rendering, you can disable it (per list).
"Archive Rendering Mode" for this list is already set to "Plain text".
That's probably something Abhilash Raj <maxking@asynchronous.in> needs to look at, as I would think the mistune code should not be called at all in that case. Although the exception seems likely to be a mistune issue, this seems rather a HyperKitty bug. Mark may have some ideas, but I'm pretty sure this is Abhilash's code.
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan
Stephen J. Turnbull wrote:
Thanks ? it's version 3.1.3, so it's these lines: https://github.com/lepture/mistune/blob/v3.1.3/src/mistune/core.py#L212-L216 OK. The version bump and the assert suggest to me that they've made significant changes. I don't know much about docker, but if you can arrange for the container to be built with "mistune<3.1" in the requirements.txt file or other file specifying the Python packages,
Matt Blissett via Mailman-users writes: that might work better as I cannot recall reports of this kind of problem. (No promises, just going on the "significant changes" guess.)
Downgrading mistune as suggested has fixed the problem, thanks!
I've made a pull request for the Docker image: https://github.com/maxking/docker-mailman/pull/754
-- Matt Blissett
On 5/5/25 06:18, Matt Blissett via Mailman-users wrote:
Downgrading mistune as suggested has fixed the problem, thanks!
I've made a pull request for the Docker image: https://github.com/maxking/docker-mailman/pull/754
As I noted in the PR, the issue is https://gitlab.com/mailman/hyperkitty/-/issues/529 which is fixed for the next HyperKitty release.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Mark Sapiro -
Matt Blissett -
Stephen J. Turnbull