Dear all,
I have noticed that Ubuntu's current LTS no longer contains packages for mailman2 and its mailman3 packages are incomplete. So, prior to upgrading the server, I will have to migrate from mailman2 / distro packages to mailman3 installed either in virtualenv or docker. I have got zero experience with virtualenv and some experience with Docker. Overall, the docker method looks easier.
Is docker the default method by now?
Thanks!
Johannes
On Sat, Dec 31, 2022 at 11:15 AM Johannes Rohr johannes@rohr.org wrote:
Dear all,
I have noticed that Ubuntu's current LTS no longer contains packages for mailman2 and its mailman3 packages are incomplete. So, prior to upgrading the server, I will have to migrate from mailman2 / distro packages to mailman3 installed either in virtualenv or docker. I have got zero experience with virtualenv and some experience with Docker. Overall, the docker method looks easier.
Is docker the default method by now? mailman-users-leave@mailman3.org
The default method is virtualenv, IMHO. I rarely see talk about Docker, but it also doesn't matter which option you choose to go with as long as you are able to run the core and any other bits that you require. I think at the end of the day you still end up using the settings.py and config files, albeit in a convoluted way :)
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
I’ve used docker on my setup. I have been wondering about this though, it causes me to need to run an AWS VM continuously to run my mailing lists, and that probably isn’t really necessary. I do sometimes wonder whether docker might be a better approach and run it under ECS, so I no longer have to look after a VM.
Even better, MM3 is just a bunch of python functions at the end of the day, and it doesn’t *need* to be running continuously. Mailing lists are, more or less by definition, event driven. Has anyone considered whether it’s feasible to turn it into a bunch of Lambda functions with a serverless database behind it?
Tim
Sent from my iPhone
On 31 Dec 2022, at 08:35, Odhiambo Washington odhiambo@gmail.com wrote:
On Sat, Dec 31, 2022 at 11:15 AM Johannes Rohr johannes@rohr.org wrote:
Dear all,
I have noticed that Ubuntu's current LTS no longer contains packages for mailman2 and its mailman3 packages are incomplete. So, prior to upgrading the server, I will have to migrate from mailman2 / distro packages to mailman3 installed either in virtualenv or docker. I have got zero experience with virtualenv and some experience with Docker. Overall, the docker method looks easier.
Is docker the default method by now? mailman-users-leave@mailman3.org
The default method is virtualenv, IMHO. I rarely see talk about Docker, but it also doesn't matter which option you choose to go with as long as you are able to run the core and any other bits that you require. I think at the end of the day you still end up using the settings.py and config files, albeit in a convoluted way :)
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to tim@thecutts.org
Tim Cutts tim@thecutts.org wrote:
I’ve used docker on my setup. I have been wondering about this though, it causes me to need to run an AWS VM continuously to run my mailing lists, and that probably isn’t really necessary. I do sometimes wonder whether docker might be a better approach and run it under ECS, so I no longer have to >look after a VM.
Even better, MM3 is just a bunch of python functions at the end of the day, and it doesn’t *need* to be running continuously. Mailing lists are, more or less by definition, event driven. Has anyone considered whether it’s feasible to turn it into a bunch of Lambda functions with a serverless database behind >it?
I actually moved from Docker to Virtualenv on my machine as I wanted to use the same larger VM to run a Cloudron instance which takes over Docker. Overall I have been more happier with the virtualenv approach as it allows me to customise the install more easier. I did look at running Mailman using Azure Container instances. The main thing here is we need to get the MTA integrated, and the MTA needs to send out emails from a static IP address with reverse DNS etc setup. This isn't easy to do on anything other than a VM.
Thanks. Andrew.
On Sat, Dec 31, 2022 at 2:29 PM Andrew Hodgson andrew@hodgson.io wrote:
Tim Cutts tim@thecutts.org wrote:
I’ve used docker on my setup. I have been wondering about this though, it causes me to need to run an AWS VM continuously to run my mailing lists, and that probably isn’t really necessary. I do sometimes wonder whether docker might be a better approach and run it under ECS, so I no longer have to >look after a VM.
Even better, MM3 is just a bunch of python functions at the end of the day, and it doesn’t *need* to be running continuously. Mailing lists are, more or less by definition, event driven. Has anyone considered whether it’s feasible to turn it into a bunch of Lambda functions with a serverless database behind >it?
I actually moved from Docker to Virtualenv on my machine as I wanted to use the same larger VM to run a Cloudron instance which takes over Docker. Overall I have been more happier with the virtualenv approach as it allows me to customise the install more easier. I did look at running Mailman using Azure Container instances. The main thing here is we need to get the MTA integrated, and the MTA needs to send out emails from a static IP address with reverse DNS etc setup. This isn't easy to do on anything other than a VM.
MM3 can use a local or a remote MTA, I believe. However, for the MTA to send out emails to the "Internet", then DNS/PTR requirements must be met.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
Am 31.12.22 um 12:29 schrieb Andrew Hodgson:
I actually moved from Docker to Virtualenv on my machine as I wanted to use the same larger VM to run a Cloudron instance which takes over Docker. Overall I have been more happier with the virtualenv approach as it allows me to customise the install more easier. I did look at running Mailman using Azure Container instances. The main thing here is we need to get the MTA integrated, and the MTA needs to send out emails from a static IP address with reverse DNS etc setup. This isn't easy to do on anything other than a VM.
Sorry, can you elaborate? Are you talking about an MTA that runs inside a VM?
Docker actually isn't a VM, so I am a bit confused.
Johannes
Thanks. Andrew.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to johannes@rohr.org
Tim Cutts writes:
Has anyone considered whether it’s feasible to turn it into a bunch of Lambda functions with a serverless database behind it?
... you just did. I'm not sure if "patches welcome" is appropriate for what looks to me to be a pretty radical idea, but someone could give it a try. :-)
I don't think anybody in Mailman core has thought about it though. I don't really see the advantage for most sites. Hosts, especially VMs, are cheap, and unless you're running the lists for a major university or the like, lowest tier hosts will do.
As for "serverless databases", that depends on whether Django and SQLAlchemy support them. Mailman has a lot of code invested in those ORMs.
Steve
Sent from my iPhone
On 31 Dec 2022, at 13:22, Stephen J. Turnbull stephenjturnbull@gmail.com wrote: Tim Cutts writes:
Has anyone considered whether it’s feasible to turn it into a bunch of Lambda functions with a serverless database behind it?
... you just did. I'm not sure if "patches welcome" is appropriate for what looks to me to be a pretty radical idea, but someone could give it a try. :-)
I don't think anybody in Mailman core has thought about it though. I don't really see the advantage for most sites.
Cost and simplifying security updates, principally. The more I think about it the more I suspect it would be too radical a re-write, and it would probably simpler to use Amazon’s SES to send the mail rather than a traditional MTA. People have done simple newsletter type lists this way (https://blog.marcolancini.it/2020/blog-serverless-mailing-list/), and they’re very cheap to run because there’s no permanently running infrastructure, and for smaller sites almost certain to sit within the free tier. Nowhere near as sophisticated in capabilities as Mailman, but easier to keep secure, since most of the underlying dependencies are gone. No operating system to patch, no database upgrades.
Hosts, especially VMs, are cheap, and unless you're running the lists for a major university or the like, lowest tier hosts will do.
That was true until MM3. I had to double the size of VM I use when I went from MM2 to MM3. It’s still not huge, but you can’t easily run MM3 on a free tier VM on AWS any more, it needs too much RAM. I found 2GB is probably the practical minimum.
As for "serverless databases", that depends on whether Django and SQLAlchemy support them. Mailman has a lot of code invested in those ORMs.
Which is why I thought it would actually require a fairly drastic re-write, although of course AWS does offer Pg and MySQL relational databases as a service. That increases the cost again compared to running the DB yourself inside the same VM, but at the benefit of not needing to patch and upgrade the database any more. SQLAlchemy can work with DynamoDB but I’m sure that would have behaviours somewhat different from Postgres or MySQL, and would need some work.
I’m just musing here really.
Regards,
Tim
(PS for full disclosure - I work for AWS, but these comments are in my personal capacity)
Steve
I attempted to set up Amazon SES for our (paid) member-to-member veterinarian discussion list (run on mailman running on a Linode) and we were rejected automatically and again on appeal/explanation. (About 1000 members, fewer than 5 messages/day; can go weeks with no messages.) Mailing lists don't fit neatly into either "transactional" or "marketing" models and I suspect that because we weren't using their other services (unlike the blog you linked below) didn't help either.
If you get approved to use SES for MM delivery (or anyone else has been), I'd welcome hearing how people have successfully explained mailing lists to the (presumably) over-worked under-trained decision-making algorithms/employees at AWS.
Dave
David Brown, IT Specialist dave@aasv.org
-----Original Message----- From: Tim Cutts tim@thecutts.org Sent: Saturday, December 31, 2022 9:24 AM To: Stephen J. Turnbull stephenjturnbull@gmail.com Cc: mailman-users@mailman3.org Subject: [MM3-users] Re: Docker or virtualenv?
Sent from my iPhone
On 31 Dec 2022, at 13:22, Stephen J. Turnbull stephenjturnbull@gmail.com wrote: Tim Cutts writes:
Has anyone considered whether it’s feasible to turn it into a bunch of Lambda functions with a serverless database behind it?
... you just did. I'm not sure if "patches welcome" is appropriate for what looks to me to be a pretty radical idea, but someone could give it a try. :-)
I don't think anybody in Mailman core has thought about it though. I don't really see the advantage for most sites.
Cost and simplifying security updates, principally. The more I think about it the more I suspect it would be too radical a re-write, and it would probably simpler to use Amazon’s SES to send the mail rather than a traditional MTA. People have done simple newsletter type lists this way (https://blog.marcolancini.it/2020/blog-serverless-mailing-list/), and they’re very cheap to run because there’s no permanently running infrastructure, and for smaller sites almost certain to sit within the free tier. Nowhere near as sophisticated in capabilities as Mailman, but easier to keep secure, since most of the underlying dependencies are gone. No operating system to patch, no database upgrades.
Hosts, especially VMs, are cheap, and unless you're running the lists for a major university or the like, lowest tier hosts will do.
That was true until MM3. I had to double the size of VM I use when I went from MM2 to MM3. It’s still not huge, but you can’t easily run MM3 on a free tier VM on AWS any more, it needs too much RAM. I found 2GB is probably the practical minimum.
As for "serverless databases", that depends on whether Django and SQLAlchemy support them. Mailman has a lot of code invested in those ORMs.
Which is why I thought it would actually require a fairly drastic re-write, although of course AWS does offer Pg and MySQL relational databases as a service. That increases the cost again compared to running the DB yourself inside the same VM, but at the benefit of not needing to patch and upgrade the database any more. SQLAlchemy can work with DynamoDB but I’m sure that would have behaviours somewhat different from Postgres or MySQL, and would need some work.
I’m just musing here really.
Regards,
Tim
(PS for full disclosure - I work for AWS, but these comments are in my personal capacity)
Steve
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to dave@aasv.org
Am 31.12.22 um 09:34 schrieb Odhiambo Washington:
The default method is virtualenv, IMHO.
Thanks! I have now chosen docker, and it is not as bad as it looked. Docker comes with huge overhead. It first looks like a true turnkey solution, but it seems the installation in virtualenv is much slicker and easier to manage.
Took me some days, especially the postfix config was tricky. But now it is up and running and I have already migrated several lists, and everything seems to be quite fine.
Thanks,
Johannes
I wanted to say "I have now chosen virtualenv". Sorry for the confusion.
Am 07.01.23 um 11:51 schrieb Johannes Rohr:
Am 31.12.22 um 09:34 schrieb Odhiambo Washington:
The default method is virtualenv, IMHO.
Thanks! I have now chosen docker, and it is not as bad as it looked. Docker comes with huge overhead. It first looks like a true turnkey solution, but it seems the installation in virtualenv is much slicker and easier to manage.
Took me some days, especially the postfix config was tricky. But now it is up and running and I have already migrated several lists, and everything seems to be quite fine.
Thanks,
Johannes
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to johannes@rohr.org
On Sat, Jan 7, 2023 at 1:54 PM Johannes Rohr johannes@rohr.org wrote:
I wanted to say "I have now chosen virtualenv". Sorry for the confusion.
I have never tried docker, but soon I will. Just for the kicks.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
On 2022-12-31 09:15:38 +0100 (+0100), Johannes Rohr wrote:
I have noticed that Ubuntu's current LTS no longer contains packages for mailman2 and its mailman3 packages are incomplete. So, prior to upgrading the server, I will have to migrate from mailman2 / distro packages to mailman3 installed either in virtualenv or docker. I have got zero experience with virtualenv and some experience with Docker. Overall, the docker method looks easier. [...]
In the OpenDev Collaboratory we were using Ubuntu LTS with the packaged MM2 it included, and have similarly migrated to using container images of MM3 on Ubuntu LTS. We rebuild images based on the ones maxking maintains, but update some of the Mailman components to newer versions (those images tend to lag behind a bit) and update some included configuration to match the upstream defaults better. If you want to see, our image adjustments are at https://opendev.org/opendev/system-config/src/branch/master/docker/mailman and the Ansible we use to deploy and configure them can be found at https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/... (minus keys and credentials of course).
It seems to be working well for our needs, but we really only went the container image route because we're already deploying most of our other services in similar containers anyway and can reuse a lot of the same automation. If that's not your situation, then the virtualenv approach will probably get you something closer to upstream Mailman development.
Jeremy Stanley
Johannes Rohr writes:
Overall, the docker method looks easier.
Not sure what you mean by that. If you mean using Mailman-provided Docker images, yeah, it's almost turnkey -- until you need support. Which seems to happen frequently enough on install (network issues, mostly, non-trivial because the three apps are configured as separate containers, and communication among Mailman apps, MTAs, webservers, and WSGI providers takes place over TCP connections). :-/
If you mean that you'll configure your own Docker image and you know how to plumb through the various TCP/IP connections to communicate among the various applications and services, and get to and from your users (presumably by the public Internet), definitely, go ahead with Docker rather than venv.
But basically Docker and venv are equivalent in the sense that they are ways to isolate your Mailman installation from the system Python to a great extent. Creating a venv is a one-off, you can put bin/activate in the 'mailman' user's .profile, and after that you don't touch the venv itself until you need to upgrade Python 6 or 7 years later. It's definitely fewer commands to patch your venv in-place and hit "/etc/init.d/$APP restart", than to patch sources and rebuild and reload a docker image.
Is docker the default method by now?
Definitely not. While I would say mostly people still install via distro packages, our recommendation for people want good control over their sites and support from us is definitely venv. Abhilash maintains our Docker images on a "best effort" basis, we don't promise support (although they're always more up-to-date than LTS packages for any distro). I don't think Mark has any Docker-specific expertise, and I have no experience yet. venv currently provides the best environment for people who are new to maintaining software (specifically Mailman) beyond what apt or yum will do for you.
participants (7)
-
Andrew Hodgson -
David Brown -
Jeremy Stanley -
Johannes Rohr -
Odhiambo Washington -
Stephen J. Turnbull -
Tim Cutts