On Sat, Apr 27, 2019, at 6:22 PM, Alain Kohli wrote:
I'm running a custom image which is based on an older version of the one here: https://github.com/maxking/docker-mailman. I attached it below. But I separated postorius and hyperkitty, so hyperkitty is running in its own container. I'm deploying the image with a plain 'docker run' behind nginx. I made fulltext_index persistent now, but it didn't get populated with anything yet. I don't really have an error traceback because there is never an error thrown. The only thing with some content is uwsgi-error.log, which you can find below. I'm also still getting the "A string literal cannot contain NUL (0x00) characters." messages. I also noticed that it takes incredibly long for the webinterface to load (several minutes) even though there doesn't seem to be any process consuming notable resources apart from the minutely job.
Funnily enough, I have the exact same image deployed on a second server as well for testing. On that one everything works fine. The only difference is that on the problematic one I have a lot more mailing lists/archives and that I imported them from mailman2. Could something have gone wrong during the import? I used the regular hyperkitty_import command.
Yes, this is because whoosh, the library set by default to run fulltext
indexing is a pure python implementation and quite slow in busy lists.
We do support more backends though, see 1 for a list of all the supported search backends. Something like Xapian(C++) or Elasticsearch/Solr(Java) should be much better in terms of performance.
uwsgi-error.log:
*** Starting uWSGI 2.0.18 (64bit) on [Sat Apr 27 22:50:17 2019] *** compiled with version: 6.4.0 on 27 April 2019 22:48:42 os: Linux-4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) nodename: hyperkitty.docker machine: x86_64 clock source: unix detected number of CPU cores: 4 current working directory: /home/hyperkitty detected binary path: /usr/local/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! setgid() to 82 setuid() to 82 chdir() to /home/hyperkitty your memory page size is 4096 bytes detected max file descriptor number: 1048576 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 0.0.0.0:8081 fd 8 uwsgi socket 1 bound to TCP address 0.0.0.0:8080 fd 9 Python version: 3.6.8 (default, Jan 30 2019, 23:54:38) [GCC 6.4.0] Python main interpreter initialized at 0x55dfaa41c980 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds [uwsgi-cron] command "./manage.py runjobs minutely" registered as cron task [uwsgi-cron] command "./manage.py runjobs quarter_hourly" registered as cron task [uwsgi-cron] command "./manage.py runjobs hourly" registered as cron task [uwsgi-cron] command "./manage.py runjobs daily" registered as cron task [uwsgi-cron] command "./manage.py runjobs monthly" registered as cron task [uwsgi-cron] command "./manage.py runjobs weekly" registered as cron task [uwsgi-cron] command "./manage.py runjobs yearly" registered as cron task mapped 208576 bytes (203 KB) for 4 cores *** Operational MODE: threaded *** WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55dfaa41c980 pid: 1 (default app) *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 1) spawned uWSGI worker 1 (pid: 40, cores: 4) Sat Apr 27 22:50:18 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 45) [uwsgi-daemons] spawning "./manage.py qcluster" (uid: 82 gid: 82) 22:50:21 [Q] INFO Q Cluster-47 starting. 22:50:21 [Q] INFO Process-1:1 ready for work at 59 22:50:21 [Q] INFO Process-1:2 ready for work at 60 22:50:21 [Q] INFO Process-1:3 ready for work at 61 22:50:21 [Q] INFO Process-1:4 ready for work at 62 22:50:21 [Q] INFO Process-1:5 monitoring at 63 22:50:21 [Q] INFO Process-1 guarding cluster at 58 22:50:21 [Q] INFO Process-1:6 pushing tasks at 64 22:50:21 [Q] INFO Q Cluster-47 running. 22:59:31 [Q] INFO Enqueued 3403 22:59:31 [Q] INFO Process-1:1 processing [update_from_mailman] 22:59:33 [Q] INFO Processed [update_from_mailman] Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 73) Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs hourly" (pid 74) [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running with pid 73 exited after 64 second(s) 23:01:28 [Q] INFO Enqueued 3404 23:01:29 [Q] INFO Process-1:2 processing [rebuild_mailinglist_cache_recent] [uwsgi-cron] command "./manage.py runjobs hourly" running with pid 74 exited after 91 second(s) Sat Apr 27 23:01:36 2019 - uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 341] during GET / (212.203.58.154) OSError: write error 23:01:36 [Q] INFO Processed [rebuild_mailinglist_cache_recent] Sat Apr 27 23:15:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 88) [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running with pid 88 exited after 4 second(s) 23:28:24 [Q] INFO Enqueued 3405 23:28:24 [Q] INFO Process-1:3 processing [update_from_mailman] 23:28:25 [Q] INFO Processed [update_from_mailman] Sat Apr 27 23:30:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 96) [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running with pid 96 exited after 4 second(s) 23:44:40 [Q] INFO Enqueued 3406 23:44:40 [Q] INFO Process-1:4 processing [update_from_mailman] 23:44:41 [Q] INFO Processed [update_from_mailman] Sat Apr 27 23:45:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 104) [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running with pid 104 exited after 4 second(s) Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 113) Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs hourly" (pid 114) Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs daily" (pid 115) Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs weekly" (pid 116) [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running with pid 113 exited after 55 second(s) [uwsgi-cron] command "./manage.py runjobs weekly" running with pid 116 exited after 55 second(s) 00:01:36 [Q] INFO Enqueued 3407 00:01:36 [Q] INFO Process-1:1 processing [rebuild_mailinglist_cache_recent] [uwsgi-cron] command "./manage.py runjobs hourly" running with pid 114 exited after 99 second(s) 00:01:50 [Q] INFO Processed [rebuild_mailinglist_cache_recent] 00:04:52 [Q] INFO Enqueued 3408 00:04:52 [Q] INFO Process-1:2 processing [update_from_mailman] 00:04:54 [Q] INFO Processed [update_from_mailman]Dockerfile:
FROM python:3.6-alpine3.7 # Add startup script to container COPY assets/docker-entrypoint.sh /usr/local/bin/ # Install packages and dependencies for hyperkitty and add user for executing apps. # It's important that the user has the UID/GID 82 so nginx can access the files. RUN set -ex \&& apk add --no-cache --virtual .build-deps gcc libc-dev linux-headers git \postgresql-dev \&& apk add --no-cache --virtual .mailman-rundeps bash sassc mailcap \postgresql-client curl \&& pip install -U django==2.2 \&& pip install git+https://gitlab.com/eestec/mailmanclient\git+https://gitlab.com/mailman/hyperkitty@c9fa4d4bfc295438d3e01cd93090064d004cf4... \git+https://gitlab.com/eestec/django-mailman3 \whoosh \uwsgi \psycopg2 \dj-database-url \typing \&& apk del .build-deps \&& addgroup -S -g 82 hyperkitty \&& adduser -S -u 82 -G hyperkitty hyperkitty \&& chmod u+x /usr/local/bin/docker-entrypoint.sh# Add needed files for uwsgi server + settings for django COPY assets/__init__.py /home/hyperkittyCOPY assets/manage.py /home/hyperkittyCOPY assets/urls.py /home/hyperkittyCOPY assets/wsgi.py /home/hyperkittyCOPY assets/uwsgi.ini /home/hyperkittyCOPY assets/settings.py /home/hyperkitty# Change ownership for uwsgi+django files and set execution rights for management script RUN chown -R hyperkitty /home/hyperkitty && chmod u+x /home/hyperkitty/manage.py# Make sure we are in the correct working dir WORKDIR /home/hyperkittyEXPOSE 8080 8081# Use stop signal for uwsgi server STOPSIGNAL SIGINTENTRYPOINT ["docker-entrypoint.sh"]CMD ["uwsgi", "--ini", "/home/hyperkitty/uwsgi.ini"]
On 4/27/19 7:58 PM, Abhilash Raj wrote:
On Sat, Apr 27, 2019, at 9:40 AM, Alain Kohli wrote:
I have run "python manage.py rebuild_index" before, doesn't that do clear_index as well? Apart from that, I run hyperkitty in a docker container and didn't know fulltext_index should be persistent, so that got deleted after every version update for sure. Which images are you using and how are you deploying them?
You should persist fulltext_index, yes, and possibly logs if you need them for debugging later.
Can you paste the entire error traceback?
On 4/26/19 10:18 PM, Mark Sapiro wrote:
On 4/26/19 11:14 AM, Alain Kohli wrote:
I see loads of "A string literal cannot contain NUL (0x00) characters." messages, but I haven't found missing messages in the archives yet. Not sure how that could be related, though. Apart from that I don't see anything unusual. The other jobs (quarter_hourly, hourly, etc.) seem to run and finish normally. Did you upgrade from a Python 2.7 version of HyperKitty to a Python 3 version? The Haystack/Whoosh search engine databases are not compatible between the two and "A string literal cannot contain NUL (0x00) characters." is the symptom.
You need to run 'python manage.py clear_index' or just remove all the files from the directory defined as 'PATH' under HAYSTACK_CONNECTIONS in your settings file (normally 'fulltext_index' in the same directory that contains your settings.py.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)