On 10/22/19 7:51 AM, Mark Dadgar wrote:
Anyone seen this before? This is the result of any search query I run. I installed ElasticSearch as the search engine yesterday and this is the result. It is still indexing (large archives).
How did you install ElasticSearch. What ElasticSearch version? This list uses elasticsearch 5.5.3 installed from the .deb as described at <https://www.elastic.co/guide/en/elasticsearch/reference/5.5/deb.html#install-deb>. You can't use a later version because there are no python bindings for versions later than 5.x In addition we use the python bindings installed with `pip install elasticsearch5` <https://pypi.org/project>/elasticsearch5/ This version also requires haystack > 2.8.1 which we installed via `pip install -e git+https://github.com/django-haystack/django-haystack.git@master#egg=django-haystack` Finally, we have these patches to the python bindings
--- elasticsearch5_backend.py.orig 2019-05-24 21:26:04.031867202 -0700 +++ elasticsearch5_backend.py 2019-05-25 08:45:42.521374647 -0700 @@ -28,6 +28,15 @@ installation of 'elasticsearch>=5.0.0,<6.0.0'. \ Please refer to the documentation." ) +# We need to override some elasticsearch_backend settings as string type is +# gone in elasticsearch 5. +from haystack.backends import elasticsearch_backend +elasticsearch_backend.DEFAULT_FIELD_MAPPING = { + "type": "text", "analyzer": "snowball"} +elasticsearch_backend.FIELD_MAPPINGS["edge_ngram"] = { + "type": "text", "analyzer": "edgengram_analyzer"} +elasticsearch_backend.FIELD_MAPPINGS["ngram"] = { + "type": "text", "analyzer": "ngram_analyzer"}
class Elasticsearch5SearchBackend(ElasticsearchSearchBackend): --- elasticsearch_backend.py.orig 2019-06-16 10:21:42.441334321 -0700 +++ elasticsearch_backend.py 2019-06-16 10:24:20.802264202 -0700 @@ -78,8 +78,10 @@ )
# Settings to add an n-gram & edge n-gram analyzer. + # And, set max_result_window. DEFAULT_SETTINGS = { "settings": { + "max_result_window": "100000", "analysis": { "analyzer": { "ngram_analyzer": {
The patch to elasticsearch_backend.py may require an even bigger value for max_result_window. The issue is the monthly Django hyperkitty - update_and_clean_index job runs into <https://stackoverflow.com/questions/41677198/result-window-is-too-large-from-size-must-be-less-than-or-equal-to-10000-b/41677310>
Mailman 3.2.1 from the debian packages in Ubuntu 19.10.
- Mark ----- mark@pdc-racing.net | 408-348-2878
Begin forwarded message:
From: root@pdc-racing.net Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/hyperkitty/search Date: October 22, 2019 at 7:44:49 AM PDT To: mark@pdc-racing.net
Internal Server Error: /mailman3/hyperkitty/search
UnboundLocalError at /hyperkitty/search local variable 'search_exception' referenced before assignment
Request Method: GET Request URL: https://mail.pdc-racing.net/mailman3/hyperkitty/search?q=Ken ... Traceback:
File "/usr/lib/python3/dist-packages/django/core/paginator.py" in count 79. return self.object_list.count()
File "/usr/lib/python3/dist-packages/haystack/query.py" in count 522. return len(self)
File "/usr/lib/python3/dist-packages/haystack/query.py" in __len__ 86. self._result_count = self.query.get_count()
File "/usr/lib/python3/dist-packages/haystack/backends/__init__.py" in get_count 619. self.run()
File "/usr/lib/python3/dist-packages/haystack/backends/elasticsearch_backend.py" in run 951. results = self.backend.search(final_query, **search_kwargs)
File "/usr/lib/python3/dist-packages/haystack/backends/__init__.py" in wrapper 33. return func(obj, query_string, *args, **kwargs)
File "/usr/lib/python3/dist-packages/haystack/backends/elasticsearch_backend.py" in search 524. _source=True)
File "/usr/local/lib/python3.7/dist-packages/elasticsearch/client/utils.py" in _wrapped 84. return func(*args, params=params, **kwargs)
During handling of the above exception (search() got an unexpected keyword argument 'doc_type'), another exception occurred:
I suspect this error is because indexing is still in process. I'm not certain, but I think so.
File "/usr/lib/python3/dist-packages/hyperkitty/views/search.py" in search 90. request.GET.get('count'),
File "/usr/lib/python3/dist-packages/django_mailman3/lib/paginator.py" in paginate 72. objects = paginator.page(page_num)
File "/usr/lib/python3/dist-packages/django/core/paginator.py" in page 57. number = self.validate_number(number)
File "/usr/lib/python3/dist-packages/django/core/paginator.py" in validate_number 46. if number > self.num_pages:
File "/usr/lib/python3/dist-packages/django/utils/functional.py" in __get__ 35. res = instance.__dict__[self.name] = self.func(instance)
File "/usr/lib/python3/dist-packages/django/core/paginator.py" in num_pages 91. if self.count == 0 and not self.allow_empty_first_page:
File "/usr/lib/python3/dist-packages/django/utils/functional.py" in __get__ 35. res = instance.__dict__[self.name] = self.func(instance)
File "/usr/lib/python3/dist-packages/django/core/paginator.py" in count 84. return len(self.object_list)
File "/usr/lib/python3/dist-packages/haystack/query.py" in __len__ 86. self._result_count = self.query.get_count()
File "/usr/lib/python3/dist-packages/haystack/backends/__init__.py" in get_count 619. self.run()
File "/usr/lib/python3/dist-packages/haystack/backends/elasticsearch_backend.py" in run 951. results = self.backend.search(final_query, **search_kwargs)
File "/usr/lib/python3/dist-packages/haystack/backends/__init__.py" in wrapper 33. return func(obj, query_string, *args, **kwargs)
File "/usr/lib/python3/dist-packages/haystack/backends/elasticsearch_backend.py" in search 524. _source=True)
File "/usr/local/lib/python3.7/dist-packages/elasticsearch/client/utils.py" in _wrapped 84. return func(*args, params=params, **kwargs)
During handling of the above exception (search() got an unexpected keyword argument 'doc_type'), another exception occurred:
File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py" in inner 41. response = get_response(request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response 187. response = self.process_exception_by_middleware(e, request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py" in _get_response 185. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/search.py" in search 100. if not isinstance(e, search_exception):
Exception Type: UnboundLocalError at /hyperkitty/search Exception Value: local variable 'search_exception' referenced before assignment
This is a bug in HyperKitty. We've never encountered it, but the code at <https://gitlab.com/mailman/hyperkitty/blob/master/hyperkitty/views/search.py#L93> only considers the whoosh and xapian backends and needs to be updated for elasticsearch. I've reported that at <https://gitlab.com/mailman/hyperkitty/issues/263>. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan