Mailman 3.2.0 Hyperkitty fulltext index error

Hi,
I've just upgraded to Mailman 3.2.0 (finally everything Python 3 now, thanks!) and it appears to work. However, there's a repeating error message in mailman-suite's logfile for Hyperkitty that looks like this:
ERROR 2018-07-12 14:28:33,895 2742 hyperkitty.lib.utils Failed to update the fulltext index: ord() expected a character, but string of length 0 found
Traceback (most recent call last):
File "/home/mailman/env3/lib/python3.5/site-packages/hyperkitty/lib/utils.py", line 179, in run_with_lock
fn(*args, **kwargs)
File "/home/mailman/env3/lib/python3.5/site-packages/hyperkitty/search_indexes.py", line 115, in update_index
update_cmd.update_backend("hyperkitty", "default")
File "/home/mailman/env3/lib/python3.5/site-packages/haystack/management/commands/update_index.py", line 276, in update_backend
last_max_pk=max_pk)
File "/home/mailman/env3/lib/python3.5/site-packages/haystack/management/commands/update_index.py", line 97, in do_update
backend.update(index, current_qs, commit=commit)
File "/home/mailman/env3/lib/python3.5/site-packages/haystack/backends/whoosh_backend.py", line 178, in update
self.setup()
File "/home/mailman/env3/lib/python3.5/site-packages/haystack/backends/whoosh_backend.py", line 126, in setup
self.index = self.storage.open_index(schema=self.schema)
File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/filedb/filestore.py", line 176, in open_index
return indexclass(self, schema=schema, indexname=indexname)
File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/index.py", line 632, in read
check_size("int", _INT_SIZE)
File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/index.py", line 626, in check_size
sz = stream.read_varint()
File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/filedb/structfile.py", line 191, in read_varint
return read_varint(self.read)
File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/util/varints.py", line 102, in read_varint
b = ord(readfn(1))
TypeError: ord() expected a character, but string of length 0 found
The error repeats every minute, so I suppose it comes from the minutely cronjob. This cronjob looks like this:
- mailman bash -c 'cd /home/mailman && . /home/mailman/env3/bin/activate && django-admin runjobs minutely --pythonpath /home/mailman/mailman-suite/mailman-suite_project --settings settings'
What's wrong here?
Marvin

On 7/12/18 7:34 AM, Marvin Gülker wrote:
Hi,
I've just upgraded to Mailman 3.2.0 (finally everything Python 3 now, thanks!) and it appears to work. However, there's a repeating error message in mailman-suite's logfile for Hyperkitty that looks like this:
ERROR 2018-07-12 14:28:33,895 2742 hyperkitty.lib.utils Failed to update the fulltext index: ord() expected a character, but string of length 0 found Traceback (most recent call last): File "/home/mailman/env3/lib/python3.5/site-packages/hyperkitty/lib/utils.py", line 179, in run_with_lock fn(*args, **kwargs) File "/home/mailman/env3/lib/python3.5/site-packages/hyperkitty/search_indexes.py", line 115, in update_index update_cmd.update_backend("hyperkitty", "default") File "/home/mailman/env3/lib/python3.5/site-packages/haystack/management/commands/update_index.py", line 276, in update_backend last_max_pk=max_pk) File "/home/mailman/env3/lib/python3.5/site-packages/haystack/management/commands/update_index.py", line 97, in do_update backend.update(index, current_qs, commit=commit) File "/home/mailman/env3/lib/python3.5/site-packages/haystack/backends/whoosh_backend.py", line 178, in update self.setup() File "/home/mailman/env3/lib/python3.5/site-packages/haystack/backends/whoosh_backend.py", line 126, in setup self.index = self.storage.open_index(schema=self.schema) File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/filedb/filestore.py", line 176, in open_index return indexclass(self, schema=schema, indexname=indexname) File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/index.py", line 421, in __init__ TOC.read(self.storage, self.indexname, schema=self._schema) File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/index.py", line 632, in read check_size("int", _INT_SIZE) File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/index.py", line 626, in check_size sz = stream.read_varint() File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/filedb/structfile.py", line 191, in read_varint return read_varint(self.read) File "/home/mailman/env3/lib/python3.5/site-packages/whoosh/util/varints.py", line 102, in read_varint b = ord(readfn(1)) TypeError: ord() expected a character, but string of length 0 found
The error repeats every minute, so I suppose it comes from the minutely cronjob. This cronjob looks like this:
This is due the the change from Python 2.7 to Python 3. The existing search index is not compatible with Python 3 whoosh.
I fixed this by deleting all the files in the index directory which is defined as PATH in HAYSTACK_CONNECTIONS in your settings and running 'django_admin update_index', but just running 'django_admin rebuild_index' should do it.

Hi,
Am 12. Juli 2018 um 08:14 Uhr -0700 schrieb Mark Sapiro:
This is due the the change from Python 2.7 to Python 3. The existing search index is not compatible with Python 3 whoosh.
Ah, I see.
I fixed this by deleting all the files in the index directory which is defined as PATH in HAYSTACK_CONNECTIONS in your settings and running 'django_admin update_index', but just running 'django_admin rebuild_index' should do it.
I simply followed your first suggestion, and it worked. I deleted all files in the index directory (which was "fulltext_index" in the "mailman-suite_project" directory, as we use https://gitlab.com/mailman/mailman-suite/tree/master/mailman-suite_project) and then ran update_index. That took a while, but succeeded. Since then the errors are gone.
Thank you! Marvin

for me, it's been enough to just delete the files, the minutely job has rebuilt them. Root cause seems to have been a server reboot that didn't wait for the mailman processes to finish correctly.
participants (3)
-
Jörg Schulz
-
Mark Sapiro
-
Marvin Gülker