Hi,

So, the minutely job updates the fulltext index, which uses a Python based indexing library called Whoosh. Whoosh is good for testing but doesn't work well for prod environments at scale.

There is a need to update the docs, but it would be great to move to a better indexing engine like Xapian or ElasticSearch (which we have been using on this list for a while).

As a short-term fix, you can move the minutely cron job to run every 15minutes or even 1 hour. The only delay you will see is that it will take new emails upto 1hour to be indexed via search, which I don't think is a big deal.

I tried to switch to Xapian, but hyperkitty's update of the index fails now: ERROR 2020-02-10 22:35:24,787 25592 hyperkitty.lib.utils Failed to update the fulltext index: update() got an unexpected keyword argument 'commit' Traceback (most recent call last): File "/usr/lib/python3/dist-packages/hyperkitty/lib/utils.py", line 179, in run_with_lock fn(*args, **kwargs) File "/usr/lib/python3/dist-packages/hyperkitty/search_indexes.py", line 115, in update_index update_cmd.update_backend("hyperkitty", "default") File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 276, in update_backend last_max_pk=max_pk) File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 97, in do_update backend.update(index, current_qs, commit=commit) TypeError: update() got an unexpected keyword argument 'commit'

I didn't find any further relevant configuration options in https://github.com/notanumber/xapian-haystack/tree/master

Kind Regards Bjoern

Hi Mark,

How did you install Xapian? Did you follow all the Installation and Configuration instructions at <https://github.com/notanumber/xapian-haystack/blob/master/README.rst>?

I installed the corresponding Debian packages: python3-xapian-haystack, python3-xapian, libxapian30.

Configuration: HAYSTACK_CONNECTIONS = { 'default': { #'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine', 'PATH': '/var/lib/mailman3/web/fulltext_index', # You can also use the Xapian engine, it's faster and more accurate, # but requires another library. # http://django-haystack.readthedocs.io/en/v2.4.1/installing_search_engines.ht... # Example configuration for Xapian: 'ENGINE': 'xapian_backend.XapianEngine' }, }

Did you run the Django Admin rebuild_index command after installing and setting up Xapian?

rebuild_index fails with the same error: [ERROR/MainProcess] Failed indexing 1 - 1000 (retry 5/5): update() got an unexpected keyword argument 'commit' (pid 28576): update() got an unexpected keyword argument 'commit'

Best Regards Bjoern

On 2/12/20 4:51 AM, Bjoern Franke wrote:

Hi,

...

I installed the corresponding Debian packages: python3-xapian-haystack, python3-xapian, libxapian30.

some digging: The commit keyword was removed in d3f1e011da3d9bebd88c78fe7a87cd6171ae650c[1] which was included in 2.1.0

• the last release of xapian-haystack. Later this commit was reverted in 253c4c2898f16a5dee2c99324621a14e3e2a789b - so it's fixed only in master of xapian-haystack.

Which is why you should install via the recommended

pip install git+https://github.com/notanumber/xapian-haystack.git

rather than the Debian package. It would be good to report this issue to Debian so they fix their broken package.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan