Permission denied for fulltext_index
Hi,
when trying to reattach a message thread in the Hyperkitty web interface, the following error occurs:
Internal Server Error: /archives/list/testlist@lists.eden.one/thread/T6MCILAVQYV5QIOGHJB3JNJ26Z7E2Z35/reattach-suggest
PermissionError at /archives/list/testlist@lists.eden.one/thread/T6MCILAVQYV5QIOGHJB3JNJ26Z7E2Z35/reattach-suggest [Errno 13] Permission denied: 'fulltext_index'
The error occurs while haystack attempts to create a directory –
File "/opt/mailman/venv/lib/python3.10/site-packages/haystack/backends/whoosh_backend.py", line 412, in search self.setup() File "/opt/mailman/venv/lib/python3.10/site-packages/haystack/backends/whoosh_backend.py", line 133, in setup os.makedirs(self.path) File "/usr/lib/python3.10/os.py", line 225, in makedirs mkdir(name, mode)
Exception Type: PermissionError at /archives/list/testlist@lists.eden.one/thread/T6MCILAVQYV5QIOGHJB3JNJ26Z7E2Z35/reattach-suggest Exception Value: [Errno 13] Permission denied: 'fulltext_index' Raised during: hyperkitty.views.thread.reattach_suggest Request information: USER: mailman
– but the existing directory /opt/mailman/fulltext_index (and its contents) are readable and writeable by mailman:mailman:
drwxrwxr-x 2 mailman mailman 4096 Nov 27 08:00 fulltext_index/
-rw-rw-r-- 1 mailman mailman 4473 Nov 27 08:00 _MAIN_65.toc -rw-rw-r-- 1 mailman mailman 15456 Nov 27 08:00 MAIN_bc0cs2aohovftat6.seg -rw-rw-r-- 1 mailman mailman 15454 Nov 27 06:00 MAIN_brbo8i8edmlz0za4.seg -rw-rw-r-- 1 mailman mailman 15462 Nov 27 07:00 MAIN_ewnemc7lsnodd3y7.seg -rw-rw-r-- 1 mailman mailman 157324 Nov 27 04:00 MAIN_qag60g1rrumshtc2.seg -rw-rw-r-- 1 mailman mailman 15459 Nov 27 05:00 MAIN_w6e0ohurasusy61d.seg -rwxrwxr-x 1 mailman mailman 0 Nov 24 16:00 MAIN_WRITELOCK*
I fail to see where haystack tries (and fails) to create the directory. FWIW, deleting threads is not a problem.
Thanks, Jan
On 11/27/22 00:35, Jan Eden via Mailman-users wrote:
– but the existing directory /opt/mailman/fulltext_index (and its contents) are readable and writeable by mailman:mailman:
drwxrwxr-x 2 mailman mailman 4096 Nov 27 08:00 fulltext_index/
-rw-rw-r-- 1 mailman mailman 4473 Nov 27 08:00 _MAIN_65.toc -rw-rw-r-- 1 mailman mailman 15456 Nov 27 08:00 MAIN_bc0cs2aohovftat6.seg -rw-rw-r-- 1 mailman mailman 15454 Nov 27 06:00 MAIN_brbo8i8edmlz0za4.seg -rw-rw-r-- 1 mailman mailman 15462 Nov 27 07:00 MAIN_ewnemc7lsnodd3y7.seg -rw-rw-r-- 1 mailman mailman 157324 Nov 27 04:00 MAIN_qag60g1rrumshtc2.seg -rw-rw-r-- 1 mailman mailman 15459 Nov 27 05:00 MAIN_w6e0ohurasusy61d.seg -rwxrwxr-x 1 mailman mailman 0 Nov 24 16:00 MAIN_WRITELOCK*
I fail to see where haystack tries (and fails) to create the directory. FWIW, deleting threads is not a problem.
It is uwsgi trying to do this. What user is it running as? Maybe you
need to add that user to the mailman group.
Deleting threads is not an issue, apparently because it doesn't try to update the search index.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-11-27 08:01, Mark Sapiro wrote:
On 11/27/22 00:35, Jan Eden via Mailman-users wrote:
– but the existing directory /opt/mailman/fulltext_index (and its contents) are readable and writeable by mailman:mailman:
drwxrwxr-x 2 mailman mailman 4096 Nov 27 08:00 fulltext_index/
-rw-rw-r-- 1 mailman mailman 4473 Nov 27 08:00 _MAIN_65.toc -rw-rw-r-- 1 mailman mailman 15456 Nov 27 08:00 MAIN_bc0cs2aohovftat6.seg -rw-rw-r-- 1 mailman mailman 15454 Nov 27 06:00 MAIN_brbo8i8edmlz0za4.seg -rw-rw-r-- 1 mailman mailman 15462 Nov 27 07:00 MAIN_ewnemc7lsnodd3y7.seg -rw-rw-r-- 1 mailman mailman 157324 Nov 27 04:00 MAIN_qag60g1rrumshtc2.seg -rw-rw-r-- 1 mailman mailman 15459 Nov 27 05:00 MAIN_w6e0ohurasusy61d.seg -rwxrwxr-x 1 mailman mailman 0 Nov 24 16:00 MAIN_WRITELOCK*
I fail to see where haystack tries (and fails) to create the directory. FWIW, deleting threads is not a problem.
It is uwsgi trying to do this. What user is it running as? Maybe you need to add that user to the
mailmangroup.
That's what baffles me – uwsgi is run as the mailman user:
mailman 3904667 1.3 0.5 129516 62188 ? Ss 18:33 0:00 /opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini mailman 3904668 0.0 0.3 203248 48796 ? Sl 18:33 0:00 /opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini mailman 3904670 0.8 0.6 220436 74464 ? Sl 18:33 0:00 /opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini
- Jan
On 2022-11-27 19:36, Jan Eden wrote:
On 2022-11-27 08:01, Mark Sapiro wrote:
On 11/27/22 00:35, Jan Eden via Mailman-users wrote:
– but the existing directory /opt/mailman/fulltext_index (and its contents) are readable and writeable by mailman:mailman:
drwxrwxr-x 2 mailman mailman 4096 Nov 27 08:00 fulltext_index/
-rw-rw-r-- 1 mailman mailman 4473 Nov 27 08:00 _MAIN_65.toc -rw-rw-r-- 1 mailman mailman 15456 Nov 27 08:00 MAIN_bc0cs2aohovftat6.seg -rw-rw-r-- 1 mailman mailman 15454 Nov 27 06:00 MAIN_brbo8i8edmlz0za4.seg -rw-rw-r-- 1 mailman mailman 15462 Nov 27 07:00 MAIN_ewnemc7lsnodd3y7.seg -rw-rw-r-- 1 mailman mailman 157324 Nov 27 04:00 MAIN_qag60g1rrumshtc2.seg -rw-rw-r-- 1 mailman mailman 15459 Nov 27 05:00 MAIN_w6e0ohurasusy61d.seg -rwxrwxr-x 1 mailman mailman 0 Nov 24 16:00 MAIN_WRITELOCK*
I fail to see where haystack tries (and fails) to create the directory. FWIW, deleting threads is not a problem.
It is uwsgi trying to do this. What user is it running as? Maybe you need to add that user to the
mailmangroup.That's what baffles me – uwsgi is run as the mailman user:
mailman 3904667 1.3 0.5 129516 62188 ? Ss 18:33 0:00 /opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini mailman 3904668 0.0 0.3 203248 48796 ? Sl 18:33 0:00 /opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini mailman 3904670 0.8 0.6 220436 74464 ? Sl 18:33 0:00 /opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini
I tried setting permissions on /opt/mailman/fulltext_index to 777 temporarily, but the issue remained (as expected). It occurs in the Whoosh backend of Haystack (whoosh_backend.py, line 132-133):
if self.use_file_storage and not os.path.exists(self.path):
os.makedirs(self.path)where self.path is referenced, which is defined in line 110 (as part of the __init__ function):
def __init__(self, connection_alias, **connection_options): super().__init__(connection_alias, **connection_options) self.setup_complete = False self.use_file_storage = True self.post_limit = getattr(connection_options, "POST_LIMIT", 128 * 1024 * 1024) self.path = connection_options.get("PATH")
Unfortunately, I cannot see where the PATH key for connection_options is defined.
- Jan
On 11/30/22 05:32, Jan Eden via Mailman-users wrote:
I tried setting permissions on /opt/mailman/fulltext_index to 777 temporarily, but the issue remained (as expected).
Are you running some security manager (SELinux, apparmor, ???)
I suspect that's the issue.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-11-30 06:52, Mark Sapiro wrote:
On 11/30/22 05:32, Jan Eden via Mailman-users wrote:
I tried setting permissions on /opt/mailman/fulltext_index to 777 temporarily, but the issue remained (as expected).
Are you running some security manager (SELinux, apparmor, ???)
I suspect that's the issue.
apparmor is disabled, and the error is only triggered when the directory does not exist:
# Make sure the index is there. if self.use_file_storage and not os.path.exists(self.path): os.makedirs(self.path) new_index = True
/opt/mailman/fulltext_index exists (and is populated), so I must assume that the Whoosh backend uses some other (non-writeable) path at this point.
- Jan
Jan Eden wrote:
I tried setting permissions on /opt/mailman/fulltext_index to 777 temporarily, but the issue remained (as expected). It occurs in the Whoosh backend of Haystack (whoosh_backend.py, line 132-133):
if self.use_file_storage and not os.path.exists(self.path): os.makedirs(self.path)where self.path is referenced, which is defined in line 110 (as part of the __init__ function):
def __init__(self, connection_alias, **connection_options): super().__init__(connection_alias, **connection_options) self.setup_complete = False self.use_file_storage = True self.post_limit = getattr(connection_options, "POST_LIMIT", 128 * 1024 * 1024) self.path = connection_options.get("PATH")
Unfortunately, I cannot see where the PATH key for connection_options is defined.
It is defined in the HAYSTACK_CONNECTIONS dictionary in your Django settings.
On 2022-11-30 16:25, Mark Sapiro wrote:
Jan Eden wrote:
I tried setting permissions on /opt/mailman/fulltext_index to 777 temporarily, but the issue remained (as expected). It occurs in the Whoosh backend of Haystack (whoosh_backend.py, line 132-133):
if self.use_file_storage and not os.path.exists(self.path): os.makedirs(self.path)where self.path is referenced, which is defined in line 110 (as part of the __init__ function):
def __init__(self, connection_alias, **connection_options): super().__init__(connection_alias, **connection_options) self.setup_complete = False self.use_file_storage = True self.post_limit = getattr(connection_options, "POST_LIMIT", 128 * 1024 * 1024) self.path = connection_options.get("PATH")
Unfortunately, I cannot see where the PATH key for connection_options is defined.
It is defined in the HAYSTACK_CONNECTIONS dictionary in your Django settings.
The dictionary contains the following:
HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'haystack.backends.whoosh_backend.WhooshEngine', 'PATH': "fulltext_index", }, }
Changing the PATH value to a full path (/opt/mailman/fulltext_index) and restarting mailman-web generates a different error:
ERROR 2022-11-30 16:38:37,953 1028315 django.request Internal Server Error: /archives/list/testing@lists.eden.one/thread/T6MCILAVQYV5QIOGHJB3JNJ26Z7E2Z35/reattach-suggest Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/lib/view_helpers.py", line 137, in inner return func(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/views/thread.py", line 454, in reattach_suggest sugg_thread = msg.object.thread AttributeError: 'NoneType' object has no attribute 'thread'
- Jan
On 11/30/22 08:41, Jan Eden via Mailman-users wrote:
Changing the PATH value to a full path (/opt/mailman/fulltext_index) and restarting mailman-web generates a different error:
Yes, it should be a full path.
ERROR 2022-11-30 16:38:37,953 1028315 django.request Internal Server Error: /archives/list/testing@lists.eden.one/thread/T6MCILAVQYV5QIOGHJB3JNJ26Z7E2Z35/reattach-suggest Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/lib/view_helpers.py", line 137, in inner return func(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/views/thread.py", line 454, in reattach_suggest sugg_thread = msg.object.thread AttributeError: 'NoneType' object has no attribute 'thread'
This says that searching the archive for this list for messages matching the current message subject returns None. It is probably a bug that we don't test for this, but what exactly are you doing? See the post at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... for the only way I've been able to create an error like this.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-11-30 10:22, Mark Sapiro wrote:
On 11/30/22 08:41, Jan Eden via Mailman-users wrote:
Changing the PATH value to a full path (/opt/mailman/fulltext_index) and restarting mailman-web generates a different error:
Yes, it should be a full path.
I encountered yet another problem with emails not getting archived anymore:
ERROR 2022-12-01 09:18:00,111 1337915 hyperkitty.views.mailman Access to the archiving API endpoint was forbidden from IP XXX.XXX.XXX.XXX, your MAILMAN_ARCHIVER_FROM setting may be misconfigured
where XXX.XXX.XXX.XXX is my server's public IP. This was probably caused by the switch to uwsgi_pass/a file socket, and could be fixed by adding
MAILMAN_ARCHIVER_FROM = ('XXX.XXX.XXX.XXX', '::1')
to /etc/mailman3/settings.py.
ERROR 2022-11-30 16:38:37,953 1028315 django.request Internal Server Error: /archives/list/testing@lists.eden.one/thread/T6MCILAVQYV5QIOGHJB3JNJ26Z7E2Z35/reattach-suggest Traceback (most recent call last): File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner response = get_response(request) File "/opt/mailman/venv/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/lib/view_helpers.py", line 137, in inner return func(request, *args, **kwargs) File "/opt/mailman/venv/lib/python3.10/site-packages/hyperkitty/views/thread.py", line 454, in reattach_suggest sugg_thread = msg.object.thread AttributeError: 'NoneType' object has no attribute 'thread'
This says that searching the archive for this list for messages matching the current message subject returns None. It is probably a bug that we don't test for this, but what exactly are you doing? See the post at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... for the only way I've been able to create an error like this.
There are currently 5 (test) threads in the archive. After changing the PATH in the HAYSTACK_CONNECTIONS dict to /opt/mailman/fulltext_index, and fixing the issue with MAILMAN_ARCHIVER_FROM above, /opt/mailman/web/logs/mailmanweb.log does not display new error messages, but I do not get any suggestions for reattaching threads, and a search for a specific thread subject ("Third test") does not return any results.
So reattachment works in principle, but the suggest/search function does not.
- Jan
On 12/1/22 01:43, Jan Eden via Mailman-users wrote:
There are currently 5 (test) threads in the archive. After changing the PATH in the HAYSTACK_CONNECTIONS dict to /opt/mailman/fulltext_index, and fixing the issue with MAILMAN_ARCHIVER_FROM above, /opt/mailman/web/logs/mailmanweb.log does not display new error messages, but I do not get any suggestions for reattaching threads, and a search for a specific thread subject ("Third test") does not return any results.
By "search for a specific thread subject", do you mean searching via the search box at the top of HyperKitty pages? This relies on the search index for Haystack (Whoosh in your case). This index is updated by the HyperKitty update_index job. Assuming you have set up the cron jobs for Django https://docs.mailman3.org/en/latest/install/virtualenv.html#cron-jobs-for-ma..., this will be run hourly and posts added to the archive in the last hour won't be indexed.
So reattachment works in principle, but the suggest/search function does not.
Reattachment suggest doesn't depend on the search index, but it does depend on loosely matching thread subjects.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Jan Eden -
Mark Sapiro