Long-shot cry for help
Hi,
I'm having severe operational issues with my mailman3 site. My site is built on standard Ubuntu packages (18.04), and they're ancient. I'm having two big issues...
- I have the old smiley problem that I've had from the very beginning and never solved. I have added 'charset': 'utf8mb4' to the relevant config file (/etc/mailman3/mailman-web.py), but that hasn't "taken", in the sense that the calls to mysql do not include the charset stuff.
/usr/share/mailman3-web/settings_local.py links to /etc/mailman3/mailman-web.py and there is no file /usr/share/mailman3-web/settings_local.pyc. I moved it out of the way hoping it'd get re-built, but it has not been in the months since.
The result of this that I get hundreds of mail messages to root (thousands, really), rapidly filling / with mail in /var/mail/root and logging in /var/log/mail.log. The mail has something like "Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/hyperkitty/api/mailman/archive" and "DataError at /hyperkitty/api/mailman/archive (1366, u"Incorrect string value: '\\xF0\\x9F\\x98\\x8A\\x0D\\x0A...' for column 'content' at row 1")"
I have tried to trace to figure out what file is _actually_ controlling the calls to mysql. There's a file called /var/lib/ucf/cache/:etc:mailman3:mailman-web.py as well, but I have no idea what that is or if it might be used.
Is there something like 'mailman info' for mailman-web so I can see what files are being used? I'd love to fix this...
- There's this one curious cron job...
- www-data [ -f /usr/bin/django-admin ] && python2.7 /usr/bin/django-admin runjobs minutely --pythonpath /usr/share/mailman3-web --settings settings
(yes, I know python2.7 is a problem... see the last paragraph below).
These jobs start piling up. They run & run & run, and eventually the machine screeches to halt when it runs out of memory and goes belly up. This is a relatively recent development. After a while, I'll have 5 or 6 or more of these all running, looking something like this:
www-data 6366 6364 99 15:00 ? 00:12:27 python2.7 /usr/bin/django-admin runjobs minutely --pythonpath /usr/share/mailman3-web --settings settings
What this means in practice right now is that I'm forced to go in and kill these on a regular basis or the machine _will_ die and people _will_ shout at me.
Does anyone have any idea why these might not be finishing? Can I comment this out completely & live happier? Any help whatsoever?
Finally, I should add that I know my current installation is fundamentally flawed because I chose to go with distro packages. I simply cannot fix it. My strong recommendation to others who follow me is that they do NOT do this. Bite the bullet and install via pip... I'm now in the process of setting up a new server that is completely based on the latest via pip installation. I have to get away from what I have or lose my sanity.
Cheers,
David
Hi,
Finally, I should add that I know my current installation is fundamentally flawed because I chose to go with distro packages. I simply cannot fix it. My strong recommendation to others who follow me is that they do NOT do this. Bite the bullet and install via pip... I'm now in the process of setting up a new server that is completely based on the latest via pip installation. I have to get away from what I have or lose my sanity.
you could throw away all the distro packages and install all things via PIP. But set
[mta] verp_probes: yes
because after 3.3.1 bounces are processed now.
Kind Regards Bjoern
David, I had alot of the same challenges you are currently seeing. Frankly, a working installation using the Ubuntu packages is seemingly impossible.
After trying that and also pip, I elected to use the Docker method. https://github.com/maxking/docker-mailman
Sanity preserved. It is very easy. Just an idea.
- Matt Alberti
Get BlueMail for Android
On Jun 3, 2021, 9:35 AM, at 9:35 AM, David Partain via Mailman-users <mailman-users@mailman3.org> wrote:
Hi,
I'm having severe operational issues with my mailman3 site. My site is built on standard Ubuntu packages (18.04), and they're ancient. I'm having two big issues...
- I have the old smiley problem that I've had from the very beginning and never solved. I have added 'charset': 'utf8mb4' to the relevant config file (/etc/mailman3/mailman-web.py), but that hasn't "taken", in the sense that the calls to mysql do not include the charset stuff.
/usr/share/mailman3-web/settings_local.py links to /etc/mailman3/mailman-web.py and there is no file /usr/share/mailman3-web/settings_local.pyc. I moved it out of the way hoping it'd get re-built, but it has not been in the months since.
The result of this that I get hundreds of mail messages to root (thousands, really), rapidly filling / with mail in /var/mail/root and logging in /var/log/mail.log. The mail has something like "Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/hyperkitty/api/mailman/archive" and "DataError at /hyperkitty/api/mailman/archive (1366, u"Incorrect string value: '\\xF0\\x9F\\x98\\x8A\\x0D\\x0A...' for column 'content' at row 1")"
I have tried to trace to figure out what file is _actually_ controlling the calls to mysql. There's a file called /var/lib/ucf/cache/:etc:mailman3:mailman-web.py as well, but I have no idea what that is or if it might be used.
Is there something like 'mailman info' for mailman-web so I can see what files are being used? I'd love to fix this...
- There's this one curious cron job...
- www-data [ -f /usr/bin/django-admin ] && python2.7 /usr/bin/django-admin runjobs minutely --pythonpath /usr/share/mailman3-web --settings settings
(yes, I know python2.7 is a problem... see the last paragraph below).
These jobs start piling up. They run & run & run, and eventually the machine screeches to halt when it runs out of memory and goes belly up. This is a relatively recent development. After a while, I'll have 5 or 6 or more of these all running, looking something like this:
www-data 6366 6364 99 15:00 ? 00:12:27 python2.7 /usr/bin/django-admin runjobs minutely --pythonpath /usr/share/mailman3-web --settings settings
What this means in practice right now is that I'm forced to go in and kill these on a regular basis or the machine _will_ die and people _will_ shout at me.
Does anyone have any idea why these might not be finishing? Can I comment this out completely & live happier? Any help whatsoever?
Finally, I should add that I know my current installation is fundamentally flawed because I chose to go with distro packages. I simply cannot fix it. My strong recommendation to others who follow me is that they do NOT do this. Bite the bullet and install via pip... I'm now in the process of setting up a new server that is completely based on the latest via pip installation. I have to get away from what I have or lose my sanity.
Cheers,
David
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
On 6/3/21 6:34 AM, David Partain via Mailman-users wrote:
Hi,
I'm having severe operational issues with my mailman3 site. My site is built on standard Ubuntu packages (18.04), and they're ancient. I'm having two big issues...
- I have the old smiley problem that I've had from the very beginning and never solved. I have added 'charset': 'utf8mb4' to the relevant config file (/etc/mailman3/mailman-web.py), but that hasn't "taken", in the sense that the calls to mysql do not include the charset stuff.
You may need to set the character set and collation for the DB or the server to utf8mb4. See https://dev.mysql.com/doc/refman/5.7/en/charset-applications.html for MySQL (I don't know about MariaDB.
/usr/share/mailman3-web/settings_local.py links to /etc/mailman3/mailman-web.py and there is no file /usr/share/mailman3-web/settings_local.pyc. I moved it out of the way hoping it'd get re-built, but it has not been in the months since.
Python3 does not create .pyc files in the same directory as the .py. They are created in a __pycache__ sub-directory. See https://www.python.org/dev/peps/pep-3147/ for details.
However, you seem to be still running a Python 2.7 version of Django et al. In this case, if it's not creating a .pyc, it is likely because it doesn't have permission. That is not a problem. If there's no .pyc, it just runs the .py.
The result of this that I get hundreds of mail messages to root (thousands, really), rapidly filling / with mail in /var/mail/root and logging in /var/log/mail.log. The mail has something like "Subject: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/hyperkitty/api/mailman/archive" and "DataError at /hyperkitty/api/mailman/archive (1366, u"Incorrect string value: '\\xF0\\x9F\\x98\\x8A\\x0D\\x0A...' for column 'content' at row 1")"
Tell your users that smiley faces and other non-base-plane unicode emojis in email subjects add no value ;)
I have tried to trace to figure out what file is _actually_ controlling the calls to mysql. There's a file called /var/lib/ucf/cache/:etc:mailman3:mailman-web.py as well, but I have no idea what that is or if it might be used.
I don't know what that is either. The underlying database access is via Django. See, e.g., https://docs.djangoproject.com/en/3.2/ref/databases/
However, it is not the MySQL queries that are the issue.
Is there something like 'mailman info' for mailman-web so I can see what files are being used? I'd love to fix this...
- There's this one curious cron job...
- www-data [ -f /usr/bin/django-admin ] && python2.7 /usr/bin/django-admin runjobs minutely --pythonpath /usr/share/mailman3-web --settings settings
(yes, I know python2.7 is a problem... see the last paragraph below).
These jobs start piling up. They run & run & run, and eventually the machine screeches to halt when it runs out of memory and goes belly up. This is a relatively recent development. After a while, I'll have 5 or 6 or more of these all running, looking something like this:
www-data 6366 6364 99 15:00 ? 00:12:27 python2.7 /usr/bin/django-admin runjobs minutely --pythonpath /usr/share/mailman3-web --settings settings
What this means in practice right now is that I'm forced to go in and kill these on a regular basis or the machine _will_ die and people _will_ shout at me.
Does anyone have any idea why these might not be finishing? Can I comment this out completely & live happier? Any help whatsoever?
You don't NEED to run this job. In the latest releases there are no minutely jobs anyway. Prior to HyperKitty 1.3.0 the update_index job was run minutely to update the search indexes with new posts, but now it's hourly.
You probably have a huge backlog of unindexed posts causing this job to run for a long time (Whoosh is particularly slow at indexing, Xapian is much better). I suggest for now, you quit running the minutely jobs and just try manually running
python2.7 /usr/bin/django-admin runjob update_index --pythonpath
/usr/share/mailman3-web --settings settings
which will probably run for hours if not days. Then after it completes, you can add that to cron to run hourly.
Finally, I should add that I know my current installation is fundamentally flawed because I chose to go with distro packages. I simply cannot fix it. My strong recommendation to others who follow me is that they do NOT do this. Bite the bullet and install via pip... I'm now in the process of setting up a new server that is completely based on the latest via pip installation. I have to get away from what I have or lose my sanity.
Good plan.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (4)
-
Bjoern Franke -
David Partain -
Mark Sapiro -
Matthew Alberti