Installing Mailman3 on Debian Bullseye
Hi,
After several times attempts I finally successfully build a (reliable enough) mailman3 instance on Debian.
I had a blog article describing how I did it. Maybe there were something you find it ridiculous, but they are all what I found important and may not be shown in the current documents.
https://blogs.slat.org/node/1012
Hope this helps.
Franklin
On 8/26/23 04:59, Franklin Weng wrote:
Hi,
After several times attempts I finally successfully build a (reliable enough) mailman3 instance on Debian.
I had a blog article describing how I did it. Maybe there were something you find it ridiculous, but they are all what I found important and may not be shown in the current documents.
https://blogs.slat.org/node/1012
Hope this helps.
Franklin
Thank you Franklin. And also of course, thank you to all the folks who work on and support mailman, as it's really wonderfully robust and excellent software.
TL;DR: we're interested in whether anyone has run into problems with mailman3 on debian bookworm.
Franklin, I'm curious as to why (at this point, with debian bookworm having been available for several months as the current stable platform) you wanted to do this. I'm asking specifically because the team I'm a part of has worked with mailman for many years (but not yet engaged with mailman3) as our main listserv answer for our users. We do have a need to spin up a new mailman instance soon, and we were planning on doing this with mailman3 on debian bookworm. If your choice to stick with bullseye was because of problems with trying to get things configured successfully on bookworm, we would definitely like to know any specific details which have created issues.
thanks so much in advance for any additional info,
~c
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to cderr@simons-rock.edu
-- Charlie Derr Director of Instructional Technology Bard College at Simon's Rock https://simons-rock.edu GPG key: https://hope.simons-rock.edu/~cderr/Charlie_Derr_public_key.txt 413-528-7344 any/all pronouns are acceptable
Hi Charlie,
charlie derr 於 2023/8/26 19:43 寫道:
On 8/26/23 04:59, Franklin Weng wrote:
Hi,
After several times attempts I finally successfully build a (reliable enough) mailman3 instance on Debian.
I had a blog article describing how I did it. Maybe there were something you find it ridiculous, but they are all what I found important and may not be shown in the current documents.
https://blogs.slat.org/node/1012
Hope this helps.
Franklin
Thank you Franklin. And also of course, thank you to all the folks who work on and support mailman, as it's really wonderfully robust and excellent software.
TL;DR: we're interested in whether anyone has run into problems with mailman3 on debian bookworm.
Franklin, I'm curious as to why (at this point, with debian bookworm having been available for several months as the current stable platform) you wanted to do this. I'm asking specifically because the team I'm a part of has worked with mailman for many years (but not yet engaged with mailman3) as our main listserv answer for our users. We do have a need to spin up a new mailman instance soon, and we were planning on doing this with mailman3 on debian bookworm. If your choice to stick with bullseye was because of problems with trying to get things configured successfully on bookworm, we would definitely like to know any specific details which have created issues.
thanks so much in advance for any additional info, ~c
The problem was reported here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1041681
AND here:
https://gitlab.com/mailman/django-mailman3/-/issues/68
Not sure which it should belong to. There are screenshots in the bug report, but some users said they couldn't open our Nextcloud instance. If you have problems opening the screenshots too you can give me a private mail I'll send pictures to you.
And - honestly I ran a mm3 instance on Debian Buster (10) a few years ago, and when I tried to upgrade my Buster to Bullseye the mm3 was totally broken. Fortunately I made a VM snapshot before upgrading so I reverted it back. However that instance was unstable too - it would run out of memory about every two weeks, so I had to setup a cron to reboot the VM every week. I have no idea what happened. It was pretty frustrated.
Then we moved all our services to another server hosts and I decided to rebuild the mm3 instances. I tried to build-remove-rebuild several times, including building it on Bullseye but got a lot of database locked errors, then tried it on Bookworm and got a broken django admin page, and finally set up a pretty stable instance on Bullseye which has run for about 6 weeks without any problem. So I decided to share the progress and what I found.
Hope it helps.
Franklin
Just mostly out of curiousity: why don't you install as venv or docker? Then it would be somewhat more independent of underlying OS and somewhat more up to date...
I also did the installation first with the Debian-packages when moving in from mailman2, but fortunately realised quite fast, that there might be a bunch of problems. Also the last years I read on this list gave me the impression, that there are quite often problems with the Debian-packaged-installation that arise while installing ... or later when upgrading the OS. I'm quite happy with docker right now (and hope that it's kept up to date for some time longer :-D ... thanks in any cases!).
Regards, Jens.
Am 26.08.23 um 14:54 schrieb Franklin Weng:
The problem was reported here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1041681
AND here:
https://gitlab.com/mailman/django-mailman3/-/issues/68
Not sure which it should belong to. There are screenshots in the bug report, but some users said they couldn't open our Nextcloud instance. If you have problems opening the screenshots too you can give me a private mail I'll send pictures to you.
And - honestly I ran a mm3 instance on Debian Buster (10) a few years ago, and when I tried to upgrade my Buster to Bullseye the mm3 was totally broken. Fortunately I made a VM snapshot before upgrading so I reverted it back. However that instance was unstable too - it would run out of memory about every two weeks, so I had to setup a cron to reboot the VM every week. I have no idea what happened. It was pretty frustrated.
Then we moved all our services to another server hosts and I decided to rebuild the mm3 instances. I tried to build-remove-rebuild several times, including building it on Bullseye but got a lot of database locked errors, then tried it on Bookworm and got a broken django admin page, and finally set up a pretty stable instance on Bullseye which has run for about 6 weeks without any problem. So I decided to share the progress and what I found.
Hope it helps.
Franklin
Hi Jens,
Jens Günther 於 2023/8/27 02:50 寫道:
Just mostly out of curiousity: why don't you install as venv or docker? Then it would be somewhat more independent of underlying OS and somewhat more up to date...
Yeah... just because either python venv or docker kicked my *** badly. I had a few services running that way and I was... screwed up frequently. Setting them up might have no problem, but maintaining them was a pain in the a**.
Not that setting up Mailman3 on Debian is exactly sunshine. As I have said, I struggled a lot and the documents are... really confusing to me.
I also did the installation first with the Debian-packages when moving in from mailman2, but fortunately realised quite fast, that there might be a bunch of problems. Also the last years I read on this list gave me the impression, that there are quite often problems with the Debian-packaged-installation that arise while installing ... or later when upgrading the OS. I'm quite happy with docker right now (and hope that it's kept up to date for some time longer :-D ... thanks in any cases!).
Regards, Jens.
I didn't find many documents about hosting mm3 (or mm2?) using docker. But at least I have a stable, reliable MM3 instance running on Debian Bullseye now. :-)
Franklin
Franklin Weng writes:
I had a blog article describing how I did it.
Thank you for mentioning this.
As far as I know, Debian packages use the system Python and the system Django. Is that correct?
I have had trouble myself with [language.en] charset: ascii myself when importing Mailman 2 lists into Mailman 3. This was an *ancient* instance (Mailman 2 installed in 2009, 21.1.15). Save yourself pain and set it to utf-8. I need to make sure before I make such a change in a public release though.
As far as I know [language.master] charset isn't relevant anymore as all out message catalogs are UTF-8 now.
I need to think about the 'django_mailman3.lib.auth.fedora' issue. I'm not sure why that should be enabled out of the box in Mailman's distribution. HyperKitty was originally a Red Hat/Fedora contribution, I guess that's why it's there and nobody has removed it. But I need to make sure before I make such a change in a public release here, too.
The utf8mb4 problem is Somebody Else's problem. We don't know what the implications of setting utf8mb4 are, that's for the MySQL folks or the Debian packager. (The fact that MySQL-family databases can't handle 4-byte UTF-8 out of the box in 2023 doesn't make me feel warm and fuzzy about MySQL, though.) Postgresql and sqlite do not have this problem.
In our current releases, the "mailman" command and the "mailman-web" command are separate. mailman-web/manage.py and django-admin are basically aliases for mailman-web. I think "mailman-web", not "mailman" is what you mean in your last note. Note that "manage.py" doesn't know anything about functions of the "mailman" command, since it is derived from Django. So you might want to check that and correct it.
Steve
Hi Steve,
Thanks for your comments!
Stephen J. Turnbull 於 2023/8/26 21:53 寫道:
Franklin Weng writes:
I had a blog article describing how I did it.
Thank you for mentioning this.
As far as I know, Debian packages use the system Python and the system Django. Is that correct?
Correct.
I have had trouble myself with [language.en] charset: ascii myself when importing Mailman 2 lists into Mailman 3. This was an *ancient* instance (Mailman 2 installed in 2009, 21.1.15). Save yourself pain and set it to utf-8. I need to make sure before I make such a change in a public release though.
As far as I know [language.master] charset isn't relevant anymore as all out message catalogs are UTF-8 now.
IIRC I got some errors about handling characters encoding if I don't set up charset in language.master and language.en section. But the memory is vague and I don't want to go back retesting that...
In our current releases, the "mailman" command and the "mailman-web" command are separate. mailman-web/manage.py and django-admin are basically aliases for mailman-web. I think "mailman-web", not "mailman" is what you mean in your last note. Note that "manage.py" doesn't know anything about functions of the "mailman" command, since it is derived from Django. So you might want to check that and correct it.
Steve
Thanks! I've updated my blog post adding your comments there.
Honestly I was totally confused by all these, and as I have said, the MM3/Django/Hyperkitty/... documents didn't help much. I struggled a lot to find out "manage.py" could do all the operations (like importing) I wanted.
Franklin
Mailman's content filtering has removed the following MIME parts from this message.
Replaced multipart/alternative part with first alternative.
Franklin Weng writes:
Thanks for your comments!
Thank you for following up on my queries.
As far as I know, Debian packages use the system Python and the system Django. Is that correct?
Correct.
This means that you're stuck with the Debian version of the Mailman 3 suite until they update which can be slow. Mailman 3 is now mature enough that if you're still happy with the Debian version a month after installing, you'll probably still be happy with it when Debian updates. It's a good tradeoff for the majority.
But for example mixing a "testing" Mailman with a "stable" Python is likely to end badly because we frequently upgrade required levels of dependencies. So you either have to get a Debian upgrade of Mailman of your current Debian version, or you need to move to a whole not-stable Debian system. Not a pleasant choice if you think you need new features or a bugfix *now*.
This is why we recommend the venv or other isolated installations. They allow you to upgrade the dependencies as well as the core packages without interfering with the rest of your system.
IIRC I got some errors about handling characters encoding if I don't set up charset in language.master and language.en section. But the memory is vague and I don't want to go back retesting that...
I suspect that you set up charset in .master and realized that it doesn't work because .en overrides it with ASCII, and then fixed .en. That's what happened to me.
Honestly I was totally confused by all these, and as I have said, the MM3/Django/Hyperkitty/... documents didn't help much.
Upstream documents are rarely very helpful with whole sets of large packages from distros in my experience. You install the packages, you go through a bunch of configuration screens, and you finish knowing *nothing about how anything works*. Then you need a little tweak that the configuration screens don't deal with, and you don't even have the vocabulary to grep or Google effectively. :-( Then again, I'm the kind of person who prepares to sleep by reading software manuals. ;-) But I do think you lose quite a bit of knowledge by using packages, and it's painful to find obscure details without it.
Django doesn't actually show its face much in Mailman, so Django docs are not much use even to the site admin, unless you actually hack on Mailman. The only exceptions are authentication and the template language. Authentication is just hard, with more details then the Lego set in the Lego store. The templating docs are pretty good, IMO.
The Mailman docs are in somewhat sad state. The content is actually quite comprehensive and detailed as these things go, but much of it is hard to discover. The original idea was to keep the docs in the sources, as doctests for examples and in the docs directory for exposition. These docs are more developer-oriented, obviously. Then there were several attempts to create a suite package which would require core, Postorius, and HyperKitty and streamline the setup process. The "mailman-suite" version came with a companion "mailman-suite-docs" package, which I think is now the basis for mailman.readthedocs.io aka docs.mailman3.org. However mailman-suite turned out to more trouble than it was worth, and mailman-web was developed to address the more useful approach of unifying some mailman-to-Django APIs for Postorius and HyperKitty (the django_mailman3 package) and configuration (a unified settings.py, as well as standardized top-level urls.py and views.py). Of course each of these contributed more documentation, some of it redundant, older versions left to slowly become obsolete (ie, incorrect) ....
"Someday" I'd like get the documents reorganized, eliminate as much redundancy as possible and make things more discoverable. But it's a mammoth task.
Steve
Hi Franklin,
On Sat, Aug 26, 2023 at 04:59:04PM +0800, Franklin Weng wrote:
I had a blog article describing how I did it. Maybe there were something you find it ridiculous, but they are all what I found important and may not be shown in the current documents.
Having been through this process with Debian 11 last year, in addition to several of your discoveries, I found that the default Whoosh full text indexer was just not usable. It is ridiculously slow and memory-hungry, and used 6+ gigabytes of RAM and days of CPU time to try to index some of my Mailman 2 mboxes.
I should think almost everyone will want to switch to Xapian instead. That is accomplished on Debian with:
# apt install libxapian30 python3-xapian python3-xapian-haystack
and then in mailman-web.py:
HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'xapian_backend.XapianEngine', 'PATH': '/var/lib/mailman3/web/fulltext_xapian_index', }, }
As far as I understand Xapian is not the default because although Xapian has an open source license, it is one that's not compatible with Mailman's GPL3 license.
At the time I installed, Debian 12 had not yet been released, and I won't upgrade that host until necessary, but I personally would not do a new install on Debian 11 so I would want to resolve the problems you had with Mailman 3 on Debian 12.
Cheers, Andy
-- https://bitfolk.com/ -- No-nonsense VPS hosting
On Wed, Aug 30, 2023 at 4:06 PM Andy Smith <andy@strugglers.net> wrote:
Hi Franklin,
On Sat, Aug 26, 2023 at 04:59:04PM +0800, Franklin Weng wrote:
I had a blog article describing how I did it. Maybe there were something you find it ridiculous, but they are all what I found important and may not be shown in the current documents.
Having been through this process with Debian 11 last year, in addition to several of your discoveries, I found that the default Whoosh full text indexer was just not usable. It is ridiculously slow and memory-hungry, and used 6+ gigabytes of RAM and days of CPU time to try to index some of my Mailman 2 mboxes.
I should think almost everyone will want to switch to Xapian instead. That is accomplished on Debian with:
# apt install libxapian30 python3-xapian python3-xapian-haystack
and then in mailman-web.py:
HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'xapian_backend.XapianEngine', 'PATH': '/var/lib/mailman3/web/fulltext_xapian_index', }, }
As far as I understand Xapian is not the default because although Xapian has an open source license, it is one that's not compatible with Mailman's GPL3 license.
At the time I installed, Debian 12 had not yet been released, and I won't upgrade that host until necessary, but I personally would not do a new install on Debian 11 so I would want to resolve the problems you had with Mailman 3 on Debian 12.
Cheers, Andy
And some unlucky admins might want to look at https://gitlab.com/mailman/hyperkitty/-/issues/408
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
participants (6)
-
Andy Smith -
charlie derr -
Franklin Weng -
Jens Günther -
Odhiambo Washington -
Stephen J. Turnbull