Internal Server Error, Need 2 values to unpack in for loop; got 1
Hello!
I am using mailman3 (version 3.3.8) and it mostly works fine, but sometimes (usually several times every day) the error below appears.
Any suggestions for what could be done about this?
Should it be considered a bug in Django, so not a mailman3 problem, or does it indicate something wrong with mailman3 itself?
Best regards, Elias
Here is what the error message looks like:
[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/postorius/lists/
Internal Server Error: /mailman3/postorius/lists/
ValueError at /postorius/lists/ Need 2 values to unpack in for loop; got 1. [...] Django Version: 3.2.19 [...] Python Version: 3.11.2 [...] Traceback (most recent call last): File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 47, in inner response = get_response(request) File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python3/dist-packages/postorius/views/list.py", line 1145, in list_index return render( File "/usr/lib/python3/dist-packages/django/shortcuts.py", line 19, in render content = loader.render_to_string(template_name, context, request, using=using) File "/usr/lib/python3/dist-packages/django/template/loader.py", line 62, in render_to_string return template.render(context, request) File "/usr/lib/python3/dist-packages/django/template/backends/django.py", line 61, in render return self.template.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 170, in render return self._render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 162, in _render return self.nodelist.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 938, in render bit = node.render_annotated(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 905, in render_annotated return self.render(context) File "/usr/lib/python3/dist-packages/django/template/loader_tags.py", line 150, in render return compiled_parent._render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 162, in _render return self.nodelist.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 938, in render bit = node.render_annotated(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 905, in render_annotated return self.render(context) File "/usr/lib/python3/dist-packages/django/template/loader_tags.py", line 62, in render result = block.nodelist.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 938, in render bit = node.render_annotated(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 905, in render_annotated return self.render(context) File "/usr/lib/python3/dist-packages/django/template/defaulttags.py", line 315, in render return nodelist.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 938, in render bit = node.render_annotated(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 905, in render_annotated return self.render(context) File "/usr/lib/python3/dist-packages/django/template/defaulttags.py", line 315, in render return nodelist.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 938, in render bit = node.render_annotated(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 905, in render_annotated return self.render(context) File "/usr/lib/python3/dist-packages/django/template/library.py", line 234, in render return t.render(new_context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 172, in render return self._render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 162, in _render return self.nodelist.render(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 938, in render bit = node.render_annotated(context) File "/usr/lib/python3/dist-packages/django/template/base.py", line 905, in render_annotated return self.render(context) File "/usr/lib/python3/dist-packages/django/template/defaulttags.py", line 203, in render raise ValueError(
Exception Type: ValueError at /postorius/lists/ Exception Value: Need 2 values to unpack in for loop; got 1. Request information: USER: AnonymousUser
[...]
On 7/11/24 11:06 AM, Elias Rudberg wrote:
Hello!
I am using mailman3 (version 3.3.8) and it mostly works fine, but sometimes (usually several times every day) the error below appears.
Any suggestions for what could be done about this? ... Here is what the error message looks like:
[Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/postorius/lists/
Internal Server Error: /mailman3/postorius/lists/
ValueError at /postorius/lists/ Need 2 values to unpack in for loop; got 1.
Can you get the page at /mailman3/postorius/lists/ and /mailman3/postorius/lists/?all-lists
If those are OK, look in your web server access logs for GET /mailman3/postorius/lists/ messages with the same timestamp as those
errors. Is there some bogus query fragment on those?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hello Mark, and thanks for answering!
Can you get the page at /mailman3/postorius/lists/ and /mailman3/ postorius/lists/?all-lists
Yes, those are OK.
If those are OK, look in your web server access logs for
GET /mailman3/ postorius/lists/messages with the same timestamp as those errors. Is there some bogus query fragment on those?
Yes, there are strange-looking long queries there at the times when these errors happen.
Here is an example of what such a request URL looks like, where I have replaced some parts with [...] to avoid including anything sensitive:
I looked up the IP address that those requests come from (all seem to come from the same IP address) and that corresponds to a hostname called something with "scanner" that looks like it belongs to some cybersecurity company. There are also many other requests from that same IP address, including things like "GET /file://etc/passwd" and similar, so it looks like someone is bombarding us with various weird requests as a form of "scanning" where they are trying to find vulnerabilities. That could also explain why we see those "Internal Server Error" errors recurring every day, if they run that vulnerability scan daily.
So maybe those strange request URLs that lead to the error messages "[Django] ERROR (EXTERNAL IP): Internal Server Error" that we are seeing, are constructed by someone who is actively trying to trigger bugs.
But anyway, no matter what strange request URL comes in, I guess the "Internal Server Error" indicates something going wrong in Mailman or in Django, a bug in how the request is processed?
/ Elias
On Fri, Jul 26, 2024 at 9:21 AM Elias Rudberg <mail@eliasrudberg.se> wrote:
So maybe those strange request URLs that lead to the error messages "[Django] ERROR (EXTERNAL IP): Internal Server Error" that we are seeing, are constructed by someone who is actively trying to trigger bugs.
I had thousands of these errors arrive in my inbox. Mine included, "NameError at /hyperkitty/list/ redacted@lists.redacted.org/thread/TBIYJOIAOIT53AJHJ3V743GZHIYZJMQW/ name 'escape_url' is not defined"
This was for the Ubuntu mailman3 packages on a fresh 24.04 LTS system, from which I imported my Mailman2.1 lists. This problem was "caused" simply by visitors to the impacted web pages (robots/spiders or human users). I could trigger the error myself just by following the problematic link.
After digging deeply into the code, and getting some support from the mailman-users list, I concluded that *the Ubuntu packages were not fit for purpose.* There were numerous issues including not being up-to-date with the official code base.
I instead followed the venv instructions ("recommended") and those problems went away.
I still am having some issues with importing Mailman 2.1 lists, and will be sending a note about that soon. Regards, Greg Newby
On 7/26/24 09:21, Elias Rudberg wrote:
Yes, there are strange-looking long queries there at the times when these errors happen.
Here is an example of what such a request URL looks like, where I have replaced some parts with [...] to avoid including anything sensitive:
I see similar ones from time to time.
I looked up the IP address that those requests come from (all seem to come from the same IP address) and that corresponds to a hostname called something with "scanner" that looks like it belongs to some cybersecurity company. There are also many other requests from that same IP address, including things like "GET /file://etc/passwd" and similar, so it looks like someone is bombarding us with various weird requests as a form of "scanning" where they are trying to find vulnerabilities. That could also explain why we see those "Internal Server Error" errors recurring every day, if they run that vulnerability scan daily.
Yes, I think these are attempts to find vulnerabilities.
So maybe those strange request URLs that lead to the error messages "[Django] ERROR (EXTERNAL IP): Internal Server Error" that we are seeing, are constructed by someone who is actively trying to trigger bugs.
I don't think this is a DOS attack or an attempt to trigger bugs. I think they are attempts to find and then exploit vulnerabilities.
But anyway, no matter what strange request URL comes in, I guess the "Internal Server Error" indicates something going wrong in Mailman or in Django, a bug in how the request is processed?
I'm not certain, but I think it's a Django issue rather than a Mailman issue. In my case, I don't get that many and I just ignore them.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
After some more digging into this I found a way to reproduce it using a much simpler URL, like the following (but with the real name instead of example.org):
https://example.org/mailman3/postorius/lists/?items=x
The error comes when Django is trying to process the template django-mailman3/django_mailman3/templates/django_mailman3/paginator/pagination.html which contains two for-loops of the following kind:
{% for key, value in request.GET.items %} [... doing something with key and value here ...] {% endfor %}
It seems like Django then gets confused by the different "items" variables involved, I think the value from the "?items=x" part of the request URL gets used instead of the "request.GET.items" (both are called "items" but they contain different things.
I don't know if the for-loops in that template could be done differently to avoid this.
There is something similar at https://github.com/django/django/blob/main/django/views/templates/technical_... where the corresponding for-loop looks like this instead:
{% for k, v in request_GET_items %} [...] {% endfor %}
So there they write "request_GET_items" instead of "request.GET.items" for some reason. I don't know if doing it like that is possible in case of the mailman template, just mentioning it because it looks like it could be another way to achieve the same thing.
I wrote a question about this in the Django forum also, here: https://forum.djangoproject.com/t/getting-error-need-2-values-to-unpack-in-f...
Hoping for an answer there to help figure out if this should be seen as a bug in Django or not.
/ Elias
Hello again,
Mark Sapiro wrote:
I'm not certain, but I think it's a Django issue rather than a Mailman issue.
I got a reply about this in the Django forum at https://forum.djangoproject.com/t/getting-error-need-2-values-to-unpack-in-f...
Quoting from that forum post:
"This is not a bug in Django. This would be functioning in the manner that Django is designed and documented at [link]"
"The error here is trying to use a reference in a template without having the view ensure that the context being passed is correct."
"Both the template and the view should be corrected here. The view should ensure the right data is put into the context, probably under a different name, and the template should render that data from that name."
(end of quote from Django forum)
So they think it is not a bug in Django.
Does it make sense, that "the template and the view should be corrected" as suggested there? (The template in question would be pagination.html but I'm not sure exactly how it should be corrected.)
/ Elias
On 7/29/24 10:23, Elias Rudberg wrote:
Does it make sense, that "the template and the view should be corrected" as suggested there? (The template in question would be pagination.html but I'm not sure exactly how it should be corrected.)
It makes sense, but it may difficult. I think we would need to somehow
sanitize the query fragments to ensure they didn't contain items=, and
if they did, probably just return an http 400 Bad Request. The
difficulty is finding all the places in all the views that need this.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Elias Rudberg -
Greg Newby -
Mark Sapiro