Op 26 jan 2024, om 22:55 heeft Mark Sapiro <mark@msapiro.net> het volgende geschreven:
On 1/26/24 02:44, Marco van Tol wrote:
Op 25 jan 2024, om 14:16 heeft Marco van Tol <mvantol@ripe.net> het volgende geschreven:
Okay, so, I got a bit further, but something still gets stuck.
Here’s what I did.
Keep in mind I’m using containers that are built from some CI/CD pipeline, so I updated the pipeline to apply the patch attached to this email to /usr/lib/python3.11/site-packages/xapian_backend.py.
Before I had 2 list servers with the “Term too long” issue, 1 got resolved by this, and the other did not.
I opened a shell in the newly deployed container to confirm the patch was applied in it.
The other attachment to this email is a copy/paste from the full error from ./manage.py rebuild_index.
Is there something else special in the email that makes it choke that evades the xapian patch?
Thank you very much in advance!
The message above with the attached "full error" never got to the list. What is the error report?
Hm, I see. Not sure why. The reply I got from mail.mailman3.org <http://mail.mailman3.org/> at 2024-01-25 13:16:25.486 UTC was:
"250 2.0.0 Ok: 12772 bytes queued as 55AFD105C02”
I’m pasting it at the bottom of this email. Sorry it didn’t come through.
I tried to change to ‘hash’, but the code in that bit of the function has not been tested enough.
For example hole = sha224(hole.encode('utf8')).hexdigest() comes back with that the bytes object hole does not have an encode() method.
When I change it to hole = sha224(hole).hexdigest(), the following error is:
That's only part of it. You need
hole = sha224(hole).hexdigest().encode('utf8')
I ended up changing it to this, which fixed that bit, and led to the next issue. :)
text = text[:match.start()] + hole + text[match.end():]
TypeError: can't concat str to bytes
The ‘hash’ part of that function needs some debugging.
Yes, presumably because no one sets XAPIAN_LONG_TERM_METHOD=hash in the environment. Do you have a reason for this?
I wanted to check and see if I could avoid the “Term too long (>245)" issue this way, but I haven’t gotten to the point where xapian is successful.
Right now I’m back at the original issue as I see no other solution than to go back to whoosh.
Thanks!
Marco van Tol
Paste:
Indexing 194620 emails
[ERROR/MainProcess] Failed indexing 156001 - 157000 (retry 5/5): Term too long (> 245): XSUBJECThttp://www.google.com/url?q=%68%74%74%70%73%3a%2f%2f%68%64%72%65%64%74%75%62... (pid 32): Term too long (> 245): XSUBJECThttp://www.google.com/url?q=%68%74%74%70%73%3a%2f%2f%68%64%72%65%64%74%75%62...
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 119, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python3.11/site-packages/xapian_backend.py", line 98, in wrapper
func(self, *args, **kwargs)
File "/usr/lib/python3.11/site-packages/xapian_backend.py", line 505, in update
database.replace_document(document_id, document)
xapian.InvalidArgumentError: Term too long (> 245): XSUBJECThttp://www.google.com/url?q=%68%74%74%70%73%3a%2f%2f%68%64%72%65%64%74%75%62...
[ERROR/MainProcess] Error updating hyperkitty using default
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 297, in handle
self.update_backend(label, using)
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 342, in update_backend
max_pk = do_update(
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 119, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python3.11/site-packages/xapian_backend.py", line 98, in wrapper
func(self, *args, **kwargs)
File "/usr/lib/python3.11/site-packages/xapian_backend.py", line 505, in update
database.replace_document(document_id, document)
xapian.InvalidArgumentError: Term too long (> 245): XSUBJECThttp://www.google.com/url?q=%68%74%74%70%73%3a%2f%2f%68%64%72%65%64%74%75%62...
Traceback (most recent call last):
File "/opt/mailman-web/./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 402, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/haystack/management/commands/rebuild_index.py", line 65, in handle
call_command("update_index", **update_options)
File "/usr/lib/python3.11/site-packages/django/core/management/__init__.py", line 198, in call_command
return command.execute(*args, **defaults)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/django/core/management/base.py", line 448, in execute
output = self.handle(*args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 297, in handle
self.update_backend(label, using)
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 342, in update_backend
max_pk = do_update(
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/haystack/management/commands/update_index.py", line 119, in do_update
backend.update(index, current_qs, commit=commit)
File "/usr/lib/python3.11/site-packages/xapian_backend.py", line 98, in wrapper
func(self, *args, **kwargs)
File "/usr/lib/python3.11/site-packages/xapian_backend.py", line 505, in update
database.replace_document(document_id, document)
xapian.InvalidArgumentError: Term too long (> 245): XSUBJECThttp://www.google.com/url?q=%68%74%74%70%73%3a%2f%2f%68%64%72%65%64%74%75%62...
