Mailman archive issues (Hyperkitty)
Hello All!
I hope its a right place to write my concern about Mailman Hyperkitty. I have same quite odd issues lately and since I'm less experienced with Mailman, any help is welcome. So i have like hybrid version pip installation, where mailman v3 is using python3 and hyperkitty is using python2 (if im correct). I also mention, that hyperkitty is a pretty old version - 1.1.4. The main issue is that if i try to open some large-scale archived list, then i get "504 Gateway Time-out" error on webUI. Any ideas, where could i start improving the situation - should i try to upgrade some packages or do i need to give more resources to host ?
Extra information:
- I have also noticed some memory violation issues which has caused by uwsgi.
- Segfaulting process - kernel: python[25870]: segfault at 24 ip 00007f9decb9a6ea sp 00007ffc413bb3d0 error 6 in libpython2.7.so.1.0[7f9decb10000+17e000]
Output from mailman.log: HyperKitty failure on http://localhost/hyperkitty/api/mailman/archive: <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx/1.16.1</center> </body> </html> (504) Exception in the HyperKitty archiver: <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx/1.16.1</center> </body> </html> (1808) Traceback (most recent call last): File "/usr/lib/python3.5/site-packages/mailman_hyperkitty/__init__.py", line 154, in _archive_message url = self._send_message(mlist, msg) File "/usr/lib/python3.5/site-packages/mailman_hyperkitty/__init__.py", line 210, in _send_message raise ValueError(result.text) ValueError: <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx/1.16.1</center> </body> </html>
On 6/2/21 12:48 AM, klillipuu@gmail.com wrote:
Hello All!
I hope its a right place to write my concern about Mailman Hyperkitty. I have same quite odd issues lately and since I'm less experienced with Mailman, any help is welcome. So i have like hybrid version pip installation, where mailman v3 is using python3 and hyperkitty is using python2 (if im correct). I also mention, that hyperkitty is a pretty old version - 1.1.4. The main issue is that if i try to open some large-scale archived list, then i get "504 Gateway Time-out" error on webUI. Any ideas, where could i start improving the situation - should i try to upgrade some packages or do i need to give more resources to host ?
It would be good to upgrade. Current HyperKitty/Postorius/Django is Python3 which will remove your dependency on Python 2.7. Also, there have been many performance improvements in more recent versions which may solve your issue.
Your gateway timeout issue is a uwsgi time limit. You can increase that in your uwsgi configuration.
Extra information:
- I have also noticed some memory violation issues which has caused by uwsgi.
- Segfaulting process - kernel: python[25870]: segfault at 24 ip 00007f9decb9a6ea sp 00007ffc413bb3d0 error 6 in libpython2.7.so.1.0[7f9decb10000+17e000]
Are these coincident with the gateway timeouts or do they occur at different times?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks for the quick replay! If its not a big trouble, could you kindly give me some advice/guide how upgrade my Hyperkitty/Postorius/Django from python2 to python3. I would be very grateful.
These segfaulting alerts appers the same time when i see "ERROR reincarnated worker Process-1:53 after death" entry in uwsgi-error.log. So it seems that they are dependent on each other.
On 6/3/21 1:06 AM, Kimmo L wrote:
Thanks for the quick replay! If its not a big trouble, could you kindly give me some advice/guide how upgrade my Hyperkitty/Postorius/Django from python2 to python3. I would be very grateful.
It depends heavily on how Mailman was installed to begin with, but essentially you just upgrade the various components to the latest versions using pip. Some things you need to be aware of are at https://docs.mailman3.org/en/latest/upgrade-3.2.html
These segfaulting alerts appers the same time when i see "ERROR reincarnated worker Process-1:53 after death" entry in uwsgi-error.log. So it seems that they are dependent on each other.
It could very well be an out of memory issue and upgrading may solve it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 03/06/2021 15:09, Mark Sapiro wrote:
On 6/3/21 1:06 AM, Kimmo L wrote:
Thanks for the quick replay! If its not a big trouble, could you kindly give me some advice/guide how upgrade my Hyperkitty/Postorius/Django from python2 to python3. I would be very grateful.
It depends heavily on how Mailman was installed to begin with, but essentially you just upgrade the various components to the latest versions using pip. Some things you need to be aware of are at https://docs.mailman3.org/en/latest/upgrade-3.2.html
If you start with python libs installed using a package manager (apt, yum, etc) and later upgrade them using pip, make sure you lock the packages at their current version so they cannot be changed by package updates, or you'll eventually have the package manager do an unintended downgrade. It is usually best to start with a distinct installation from pip, into a python venv to avoid changing system libs. You can of course seed that install with the current setup first!
Ruth
-- Software Manager & Engineer Tel: 01223 414180 Blog: http://www.ivimey.org/blog LinkedIn: http://uk.linkedin.com/in/ruthivimeycook/
Hi!
Thanks for the support and help! But so far i havent been successful with upgrading my postorius, hyperkitty and django pip packages from python2.7 to python3.5 or higher. As i understand, it was installed 2018 and has not been updated (because it has been working without any problem). Maybe the easiest way is to create a new deployment and somehow transfer data from old server to new one ?
On 6/7/21 11:54 PM, Kimmo L wrote:
Hi!
Thanks for the support and help! But so far i havent been successful with upgrading my postorius, hyperkitty and django pip packages from python2.7 to python3.5 or higher. As i understand, it was installed 2018 and has not been updated (because it has been working without any problem).
Perhaps we can help with that. How are these installed now? What issues are you having when trying to upgrade.
Maybe the easiest way is to create a new deployment and somehow transfer data from old server to new one ?
It's up to you which way to go. If you create a new deployment, I recommend following https://docs.mailman3.org/en/latest/install/virtualenv.html. You might also consider Xapian as the search engine - see https://docs.mailman3.org/en/latest/config-web.html#enable-full-text-search-... and https://github.com/notanumber/xapian-haystack/tree/master
Transferring the data is easy as long as the database is compatible - just dump and load, plus copy some files from Mailman's var/ directory.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Okay, i will try to explain the overall situation as far as I can understand: Its a CentOS 7 host without any virtualenv. Mailman_suite_project git repo has been cloned to the host and after that some packages have been installed with the help of pip:
# pip2 install django==1.10 postorius==1.1.0 hyperkitty whoosh xapian-haystack # pip3.5 install mailman mailman-hyperkitty
And now Im in trouble, what would be the right steps from here. I'm not that competent to use pip, so im confused - should i install python36 packages and try to upgrade pip2 and pip3 mailman packages after that or should i install these pip2 packages with pip3 and remove from pip2 list?
On 6/8/21 10:50 PM, Kimmo L wrote:
Okay, i will try to explain the overall situation as far as I can understand: Its a CentOS 7 host without any virtualenv. Mailman_suite_project git repo has been cloned to the host and after that some packages have been installed with the help of pip:
# pip2 install django==1.10 postorius==1.1.0 hyperkitty whoosh xapian-haystack # pip3.5 install mailman mailman-hyperkitty
For current Mailman the minimum Python version is 3.6 and all components are Python 3. Nothing is Python 2. You should be installing the latest versions of everything.
Do not install Django explicitly with pip. It will be installed as a dependency of Postorius and Hyperkitty.
If you are going to use Xapian, you don't need whoosh. Also, you need to install Xapian itself, either from source or with yum if there is a yum package.
And now Im in trouble, what would be the right steps from here. I'm not that competent to use pip, so im confused - should i install python36 packages and try to upgrade pip2 and pip3 mailman packages after that or should i install these pip2 packages with pip3 and remove from pip2 list?
There are no pip2 packages. Everything is Python >=3.6.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I have now been successful with the upgrade :) But i have one small question more about hyperkitty. I have one list, which is quite heavy loaded and contains a lot of data. Is there some optimization or cleaning possibilities, that will help to reduce loading ? (at the moment it opens that list like 5-10 minutes in hyperkitty). Or should i increase the performance of the server? (at the moment it has 1 CPU and 3gb RAM).
Kimmo L writes:
But i have one small question more about hyperkitty. I have one list, which is quite heavy loaded and contains a lot of data.
I'm not sure what you mean by "heavily loaded" (I guess the HyperKitty processes are using a lot of CPU?) or "a lot of data" (number of posts? MB?) However, 5-10 minutes to "open" the list (what do you mean by "open"? display the top page?) is not acceptable.
Beyond that, it's hard to diagnose with the information you've given, as HyperKitty has a number of options that affect performance. You can help us by doing a little research: searching in the archives for this list and in mailman-developers@python.org for "HyperKitty performance", "HyperKitty slow", and "HyperKitty speed", and doing the same in the HyperKitty issue tracker at https://gitlab.com/mailman/hyperkitty.
Also, some configuration options that are known to affect performance of some HyperKitty operations:
- The search engine. IIRC, Xapian is known to be relativeli fast, Whoosh is relatively slow.
- Configuration of caching (redis).
- Configuration of queues. When you come back with more details, you should provide full configurations for your HyperKitty installation, and tell us what your web stack (webserver, any additional Django middleware) is.
Note: You may post entire configuration files here, but you should redact sensitive information since this list is publicly archived. If you're not sure how to do that, or what is sensitive, you can negotiate sending them directly to one or more of the other developers -- I'm not very useful, I've already told you everything I know because my HyperKitty is small and just works. ;-)
Steve
Hi!
I'm very grateful for your help :)
Okay, sorry about that. I will try to give more information now. We use https://gitlab.com/mailman/mailman-suite to set up mailman with postorius and hyperkitty. Additional we use NGINX web server and postgresql database.
Settings.py configuration are basically same with this - https://gitlab.com/mailman/mailman-suite/-/blob/master/mailman-suite_project... The only differences are: *In database section we use postgresql configuration *Social accounts configuration is disabled *Haystack connections we use xapian_backend.XapianEngine engine
Here is also our mailman.cfg: [database] class: mailman.database.postgresql.PostgreSQLDatabase url: postgres://mailman:xxxxxx@localhost/mailman
[paths.custom] bin_dir: /usr/local/bin var_dir: /var/lib/mailman queue_dir: /var/spool/mailman log_dir: /var/log/mailman lock_dir: /var/lock/mailman etc_dir: /etc ext_dir: /etc/mailman.d pid_file: /var/run/mailman/master.pid
[mailman] layout: custom site_owner: xxxx anonymous_list_keep_headers:
[mta]
[loggin.archiver] path: archiver.log level: debug
[archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /etc/hyperkitty.cfg clobber_date: maybe clobber_skew: 1d
Here is our uwsgi.ini: # /etc/uwsgi.ini [uwsgi] uid = uwsgi gid = uwsgi pidfile = /run/uwsgi/uwsgi.pid emperor = /etc/uwsgi.d stats = /run/uwsgi/stats.sock chmod-socket = 660 emperor-tyrant = true cap = setgid,setuid buffer-size = 32000
# process limits limit-as = 512
# Mailman settings plugins = python36,logfile master = true process = 2 threads = 2 chdir = /opt/mailman-suite/mailman-suite_project/ wsgi-file = wsgi.py uwsgi-socket = 0.0.0.0:8000 attach-daemon = ./manage.py qcluster
# Logging req-logger = file:/var/log/uwsgi/uwsgi.log logger = cron file:/var/log/uwsgi/uwsgi-cron.log log-route = cron uwsgi-cron logger = qcluster file:/var/log/uwsgi/uwsgi-qcluster.log log-route = qcluster uwsgi-daemons logger = file:/var/log/uwsgi/uwsgi-error.log
It seems that we have some perfomance issues on webUI, if we try to check archived messages. Some lists, that contains low number of archives is not a problem but lists, that contains several thousand messages and grows faster are issue. Sometimes it loads so long, that i get "504 Gateway Time-Out" I was also monitoring server perfomance and i didnt see any issues on that side. It seems that there might be some issues to read archived messages from database. But this is just my one opinion.
I also debugged log files and found some error messages:
- mailmansuite.log -> django.db.utils.DatabaseError: out of memory for query result
it happens when i try to open a lists in webui, to check archived messages.
nginx-error.log -> upstream timed out (110: Connection timed out) while reading response header from upstream # also happens when i try to open list in hyperkitty and it takes long time to open
If any information is more needed then let me know. My one thought was also, that maybe i need to try rebuild_indexes.
On 6/15/21 11:31 AM, Kimmo L wrote:
It seems that we have some perfomance issues on webUI, if we try to check archived messages. Some lists, that contains low number of archives is not a problem but lists, that contains several thousand messages and grows faster are issue. Sometimes it loads so long, that i get "504 Gateway Time-Out" I was also monitoring server perfomance and i didnt see any issues on that side. It seems that there might be some issues to read archived messages from database. But this is just my one opinion.
I also debugged log files and found some error messages:
- mailmansuite.log -> django.db.utils.DatabaseError: out of memory for query result
it happens when i try to open a lists in webui, to check archived messages.
nginx-error.log -> upstream timed out (110: Connection timed out) while reading response header from upstream # also happens when i try to open list in hyperkitty and it takes long time to open
If any information is more needed then let me know. My one thought was also, that maybe i need to try rebuild_indexes.
Rebuilding indexes will affect only searches. It should have no effect on just opening a list. We have some large imported archives at https://mail.python.org/archives/ e.g. https://mail.python.org/archives/list/mailman-users@python.org/ with over 85,000 messages, and don't see these issues.
It definitely looks like your server doesn't have enough memory. You might also run memcached and enable it for Django https://gitlab.com/mailman/mailman-suite/-/blob/master/mailman-suite_project...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I tried memcached, but i didnt noticed big different at the moment. I tried to increase "limit-as" value in uwsgi.ini file from 512 to 1024 and it seems that the situation improved slightly. Now its shows graphs in hyperkitty tab (before it only showed loading icons) and if i try to open my biggest list (which contains around 40,000 messages) it opens but it takes a lot of time still. I increased my host CPU 1 > 2 and RAM 3 > 4GB. On the monitoring side i noticed that CPUs went 100% when i tried to open the list. Memory usage was okay.
participants (5)
-
Kimmo L -
klillipuu＠gmail.com
-
Mark Sapiro -
Ruth Ivimey-Cook -
Stephen J. Turnbull