In my quest to figure out how even us idiots can successfully install a mailman3 server, I keep looking ways to reduce the process to the simplest possible set of instructions. So far, I haven't been doing very well with that, but I've been learning a ton.
In particular, while I still haven't got a fully functioning mailman3 server yet, someone pointed me, off list, to Ansible, which I wasn't aware of before and which has completely blown me away by its power and simplicity.
For those who aren't aware of it, it's an agentless, multi-platform automation technology.
For our purposes, it replaces "how to install mailman3" guides with a single script that aims to actually do the entire thing for you. One of the particularly interesting points is that you can write complex scripts/data structures (or 'collections') that allow for a single collection to cover a broad range of use cases, such as this person's exim Ansible collection, which covers 3 very different use cases from a private domain email relay up to a full multi-domain email handler with IMAP integration.
I haven't quite got the mailman3 one working yet, but I've just started playing with it. Here are some pointers in case anyone else is interested in trying, and I'd be very interested in whether or not I'm actually the last person to discover this technology, and if one of these solutions succeeds in delivering mailman3 "out of the box".
- Install Ainsible, make sure you're on at least version 2.9, if not 2.10. Some distributions are still on 2.7, which is not compatible with more recent collections. The "Galaxy" collection is apparently only tested on Debian 10.
- Pick your mailman3 Ansible collection of choice and install it (https://galaxy.ansible.com/ucr/mailman3, or https://github.com/rivimey/ansible-mailman3) Command to install official collection - ansible-galaxy install ucr.mailman3
- Write playbook.yml with key variables (mailman3 example below)
mailman3-playbook.yml
- hosts: mailman3
vars: mailman3_domain: example.com # It will install and create a certificate using certbot, its nignx plugin and http validation (Public IP required). If false, it will install a self sign cert mailman3_install_certbot: true # install and create certificate # secrets, please change them mailman3_secret_key: "change_secretkey" mailman3_rest_api_pass: "change_api_pass" mailman3_archiver_key: "change_archiver_key" mailman3_db_pass: "change_db_pass" mailman3_web_db_pass: "change_web_db_pass" mailman3_web_admin_pass: "change_web_admin_pass" roles:- ucr.mailman3
So after a lot of experimentation, I'm quite close and wanted to share how to get a DigitalOcean Debian 10 instance (2 GB minimum) to an almost functional mailman3/postfix/postgres node.
Details of Ansible work at: https://galaxy.ansible.com/ucr/mailman3 Credit goes to: Job Céspedes Ortiz: jobcespedes@gmail.com
Outstanding issues [any corrections appreciated]:
- certbot fails for me, so I set "false" in the playbook and install certbot afterwards, which works fine
- Getting "An error occurred while processing your request." at authentication. Will try to debug this later
- I really don't understand the subtlety of when you install with root and when with the mailman user, so all of this was done with root
After creation of 2GB DigitalOcean Debian 10 instance:
apt update apt -y upgrade dpkg-reconfigure tzdata apt -y install git apt -y install curl apt -y install python3-pip python3 --version [3.7.3] curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py sudo python3 get-pip.py --user export PATH="/root/.local/bin:$PATH" pip install ansible --user ansible ansible --version [2.10.4] python3 -m pip install argcomplete sudo activate-global-python-argcomplete ansible-galaxy install ucr.mailman3 cd /home sudo nano playbook.yaml
- hosts: localhost vars: mailman3_domain: yourdomain.com # It will install and create a certificate using certbot, its nignx plugin and http validation mailman3_install_certbot: false # install and create certificate # secrets, please change them mailman3_secret_key: "change_secretkey" mailman3_rest_api_pass: "change_api_pass" mailman3_archiver_key: "change_archiver_key" mailman3_db_pass: "change_db_pass" mailman3_web_db_pass: "change_web_db_pass" mailman3_web_admin_pass: "change_web_admin_pass" roles: - ucr.mailman3
ansible-playbook -c local -i localhost, playbook.yaml
The 'authentication' error appears to come down to this, in mailman-web.log.
ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py)
Any ideas on what might cause that? More context below..
[uwsgi-daemons] respawning "python3 manage.py qcluster" (uid: 33 gid: 33) Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 206, in fetch_command klass = load_command_class(app_name, subcommand) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 40, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/django_q/management/commands/qcluster.py", line 4, in <module> from django_q.cluster import Cluster File "/usr/lib/python3/dist-packages/django_q/cluster.py", line 24, in <module> from django_q import tasks File "/usr/lib/python3/dist-packages/django_q/tasks.py", line 12, in <module> from django_q.cluster import worker, monitor ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py)
On 1/3/21 3:47 PM, ieso@johnwillson.com wrote:
Any ideas on what might cause that? More context below..
[uwsgi-daemons] respawning "python3 manage.py qcluster" (uid: 33 gid: 33) Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute() File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 206, in fetch_command klass = load_command_class(app_name, subcommand) File "/usr/lib/python3/dist-packages/django/core/management/__init__.py", line 40, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1006, in _gcd_import File "<frozen importlib._bootstrap>", line 983, in _find_and_load File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 677, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 728, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/django_q/management/commands/qcluster.py", line 4, in <module> from django_q.cluster import Cluster File "/usr/lib/python3/dist-packages/django_q/cluster.py", line 24, in <module> from django_q import tasks File "/usr/lib/python3/dist-packages/django_q/tasks.py", line 12, in <module> from django_q.cluster import worker, monitor ImportError: cannot import name 'worker' from 'django_q.cluster' (/usr/lib/python3/dist-packages/django_q/cluster.py)
This is a django_q issue. Apparently https://github.com/Koed00/django-q/issues/331, fixed by https://github.com/Koed00/django-q/pull/356
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
An attempt to use markdown to rescue the yaml code.
If this formatting fails, just cut and paste from: https://galaxy.ansible.com/ucr/mailman3
- hosts: localhost
vars:
mailman3_domain: yourdomain.com
# It will install and create a certificate using certbot, its nignx plugin and http validation
mailman3_install_certbot: false # install and create certificate
# secrets, please change them
mailman3_secret_key: "change_secretkey"
mailman3_rest_api_pass: "change_api_pass"
mailman3_archiver_key: "change_archiver_key"
mailman3_db_pass: "change_db_pass"
mailman3_web_db_pass: "change_web_db_pass"
mailman3_web_admin_pass: "change_web_admin_pass"
roles:
- ucr.mailman3
ieso@johnwillson.com writes:
For our purposes, it replaces "how to install mailman3" guides with a single script that aims to actually do the entire thing for you.
If a single script would work, we'd have one already, I'm pretty sure. The problem is that a large proportion (I'm tempted to say "almost all" ;-) Mailman 3 installations need to fit in to an existing system configuration or pre-conceived design. I'd say that that the "standard" Mailman stack is:
- Intel server box (not a VM)
- Linux kernel
- any distro
- Postfix MTA
- Apache webserver with wsgi module enabled
- PostgreSQL database manager
- Python 3 >= 3.6
- Mailman core
- Postorius
- HyperKitty
- All the above on a single host with a single public IP address and canonical domain.
So 2 and 7 brook no controversy (2 because you can't argue with "anything goes", and 7 because, well, if you're arguing about 7, you're not ready to install Mailman nohow!) But every single one of the rest of those is *very* debatable -- there are already people using all of the options mentioned below (and more) in production. VMs and containers are frequently used for 0, BSDs (including MacOS) are fine for 1, Exim4 and Sendmail have varying degrees of support already for 3 (and I'll bet there are people running qmail), people do use other webservers such as nginx for 4, aside from PostgreSQL there are the MyQSL databases that are supported and some crazy folks will try to use SQLite or Oracle in production for 5, Postorius is optional and there is a commercial alternative that is used in production in a hosting service and may eventually be available for installation on your own host for 8, ditto 8 for 9 and there are yet other options such as mailarchive.com, and for 10, some people for either security or performance reasons or both want a multihomed system.
So there are a minimum of 3 MTAs * 3 production-ready RDBMSes (my understanding is that there are Mailman config tweaks for standard MySQL that MariaDB doesn't need or vice-versa) to deal with, and possibly some optional options we'd like to deal with. The alternative is to say: here's the deal, spin up a Linux VM and run this script ... but that's basically what Abhilash's Docker install is (except the obvious that container != VM) -- and even that requires tweaks that are less than obvious to a lot of people to get the necessary network connectivity.
So I don't see Ansible (or any such system) as a panacea. Rather, I see it as a standard language that might help us in systematically tracking variables in provisioning Mailman systems. One Ansible role would be a useful reference, and if that grows into a family of roles that are easy to compare and implement "turn-key" Mailman 3 installations for a ton of users, that would be great. That's my motivation for encouraging research into it. Now, I could be wrong -- Ansible could be more powerful than I'm giving it credit for, or that "standard" configuration might not be merely what several Mailman core devs run, it might actually be the majority of systems new to Mailman. In that case just one script would be a major contribution. So don't let me discourage anybody. But do be aware that you probably do have to make a choice between a quagmire of options and mandating a particular stack (or small set of them) for simplicity.
On 6 Jan 2021, at 04:25, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote:
But do be aware that you probably do have to make a choice between a quagmire of options and mandating a particular stack (or small set of them) for simplicity.
It would be good if Mailman could do what Discourse do here. You can install it any number of ways but they push you towards one installation option. This makes it easier for and instils confidence in less technical people, and makes it easier for the developers to provide support. At the same time, it’s possible for an expert to choose different options. Have a look at: https://github.com/discourse/discourse/blob/master/docs/INSTALL-cloud.md
Best wishes
Jonathan
Jonathan M writes:
On 6 Jan 2021, at 04:25, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote:
But do be aware that you probably do have to make a choice between a quagmire of options and mandating a particular stack (or small set of them) for simplicity.
It would be good if Mailman could do what Discourse do here.
Discourse is a different kettle of fish. They concentrate on the much more straightforward (from a network and interapp communication configuration standpoint) web service because that's what their users expect. Mail is a lot more baroque from that standpoint.
We shouldn't hope to successfully "encourage" hosts to do things our way. Eg, I run Exim4 because that's what I've always done and because I'm familiar with its spam-filtering, performance-tuning, and security controls: I'm not going to switch to Postfix just because that's the most common configuration. Similar considerations apply to webserver and RDBMS (for other people; I'm comfortable with Apache and PostgreSQL). It's not just a matter of "tweaking the expert options".
As far as I can see, the best we can do is what Abhilash has already done: provide a full turnkey system in a container or a Vagrant-style VM. But that has its issues, too.
If somebody wants to prove me wrong, feel free. But my assessment is that there are much better uses of core developer time than chasing a one-size-fits-all configuration. A suite of scripts or better yet Ansible (etc) roles is a better fit for Mailman's audience, I think.
Steve
On Thu, Jan 7, 2021, at 8:18 PM, Stephen J. Turnbull wrote:
Jonathan M writes:
On 6 Jan 2021, at 04:25, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote:
But do be aware that you probably do have to make a choice between a quagmire of options and mandating a particular stack (or small set of them) for simplicity.
It would be good if Mailman could do what Discourse do here.
Discourse is a different kettle of fish. They concentrate on the much more straightforward (from a network and interapp communication configuration standpoint) web service because that's what their users expect. Mail is a lot more baroque from that standpoint.
We shouldn't hope to successfully "encourage" hosts to do things our way. Eg, I run Exim4 because that's what I've always done and because I'm familiar with its spam-filtering, performance-tuning, and security controls: I'm not going to switch to Postfix just because that's the most common configuration. Similar considerations apply to webserver and RDBMS (for other people; I'm comfortable with Apache and PostgreSQL). It's not just a matter of "tweaking the expert options".
As far as I can see, the best we can do is what Abhilash has already done: provide a full turnkey system in a container or a Vagrant-style VM. But that has its issues, too.
The containers are a good way to get Mailman Running, but it does require some additional setup including web servers and mta. There is scope for a lot of improvement in the container setup to be more "turnkey" and my time has been split between maintaining Mailman itself and $work that container setup hasn't gotten much attention beyond version updates for new releases.
If someone familiar with Containers would like to help out, there are open issues1 that could make lives of some people installing a bit easy.
If somebody wants to prove me wrong, feel free. But my assessment is that there are much better uses of core developer time than chasing a one-size-fits-all configuration. A suite of scripts or better yet Ansible (etc) roles is a better fit for Mailman's audience, I think.
Steve
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Am 08.01.21 um 05:47 schrieb Abhilash Raj:
If someone familiar with Containers would like to help out, there are open issues1 that could make lives of some people installing a bit easy.
I'm pretty comfortable with docker containers, mailman and postfix in somewhat unsual configurations, and would like to help out there (with limited time, though, but I can justify some on job work time).
What would be the most helpful contribution? I see that there are some pull requests that are not yet integrated, but didn't check whether they require additional QA to be integrated or are just backlogged. Would you rather like assistance in integrating existing pull requests, QA work to make them integration-ready, or work on untackled issues to create more pull requests? :-)
Cheers, Hans-Martin
Hans-Martin Mosner writes:
I'm pretty comfortable with docker containers, mailman and postfix in somewhat unsual configurations, and would like to help out there (with limited time, though, but I can justify some on job work time).
Cool!
What would be the most helpful contribution? I see that there are some pull requests that are not yet integrated, but didn't check whether they require additional QA to be integrated or are just backlogged.
Would you rather like assistance in integrating existing pull requests, QA work to make them integration-ready, or work on untackled issues to create more pull requests? :-)
Abhilash is busy, so in case he doesn't answer quickly, I offer my thoughts. But Abhilash he is the most important developer on the containers.
If you are considering formal membership in the core (ie, direct commit privilege, with an informal expectation of review from "more senior" core devs), integrating existing pull requests might be top priority, to get practice in that. Otherwise, reviews and triage on all container issues, and QA on existing pull requests, are probably top priorities IMO.
Steve
On Sat, Jan 9, 2021, at 9:49 AM, Stephen J. Turnbull wrote:
Hans-Martin Mosner writes:
I'm pretty comfortable with docker containers, mailman and postfix in somewhat unsual configurations, and would like to help out there (with limited time, though, but I can justify some on job work time).
Cool!
What would be the most helpful contribution? I see that there are some pull requests that are not yet integrated, but didn't check whether they require additional QA to be integrated or are just backlogged.
Would you rather like assistance in integrating existing pull requests, QA work to make them integration-ready, or work on untackled issues to create more pull requests? :-)
Abhilash is busy, so in case he doesn't answer quickly, I offer my thoughts. But Abhilash he is the most important developer on the containers.
If you are considering formal membership in the core (ie, direct commit privilege, with an informal expectation of review from "more senior" core devs), integrating existing pull requests might be top priority, to get practice in that. Otherwise, reviews and triage on all container issues, and QA on existing pull requests, are probably top priorities IMO.
Thanks Steve for the quick response!
I agree with Steve that most help would be with QA on existing issues and PRs. There aren’t many PRs pending, but some of them are pending because I am a bit weary of existing users breaking due to things like upgrading database versions etc. I don’t think there is a good way to handle those, other than just minor/major bumping versions of the release.
I’ve been debating it in my head for a while whether it makes sense to bump a minor version for a ‘sample configuration’ (docker-compose.yaml file) bumping the version of a database. Perhaps caution in documentation to verify the side-effects to simply applying the configuration from the git repo is enough?
There are a lot of tickets labelled ‘documentation’ that are rather low hanging fruits to be tackled easily. If you want to tackle something bigger, you help create a new compose configuration that adds a web server and MTA for a rather ‘complete setup’. I’ve just started labelling important issues with ‘priority p0’, so that might also be a good starting point. There aren’t many issues, but some tracker grooming/prioritization is needed too ;-)
Abhilash
Steve
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
On 8 Jan 2021, at 04:18, Stephen J. Turnbull turnbull.stephen.fw@u.tsukuba.ac.jp wrote:
Discourse is a different kettle of fish. They concentrate on the much more straightforward (from a network and interapp communication configuration standpoint) web service because that's what their users expect. Mail is a lot more baroque from that standpoint.
We shouldn't hope to successfully "encourage" hosts to do things our way.
Maybe Mail-in-a-Box is a useful comparator in terms of complexity (though it appears to be less configurable than Discourse, and the software itself is not comparable).
Maybe my experience hasn’t been typical. I’ve mainly used a shared server which didn’t have Mailman installed, but of course had email set up. It was relatively easy to install Mailman 2 but impossible to install Mailman 3. I could probably install Mailman 3 on Digital Ocean or similar, but get the feeling that the mail server is the hardest part. If I ever can set aside enough time to get it working I should write the idiot’s guide that I needed.
It would be great if there were a “Mailman-in-a-Box” including everything necessary to get Mailman working from scratch on a VPS (in addition to the primary, flexible way that allows experts to use Mailman 3 according to their personal preference).
I will look into Abhilash’s Docker container setup. I think the first time I saw that I didn’t know he was “maxking” and assumed it was a third-party thing that would likely be abandoned.
Best wishes
Jonathan
Am 08.01.21 um 12:06 schrieb Jonathan M:
It would be great if there were a “Mailman-in-a-Box” including everything necessary to get Mailman working from scratch on a VPS (in addition to the primary, flexible way that allows experts to use Mailman 3 according to their personal preference).
That would certainly be best, but the spectrum of possible environments has been widened in recent years due to containerization (which encourages splitting a service into microservices) and there would be a number of possible “Mailman-in-a-Box” variants. The simplest variant installable on a VPS should indeed be relatively straightforward, but even there you have several mail system options, several ways of running Python, several possible databases and web servers, all of which are outside of the Mailman-in-a-Box package. If you integrate with an existing postfix installation using virtual alias domains, you will need to follow the instructions at https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht... which can be a bit daunting if you don't know your postfix installation really well.
I will look into Abhilash’s Docker container setup. I think the first time I saw that I didn’t know he was “maxking” and assumed it was a third-party thing that would likely be abandoned.
This is certainly a good option, although it has still some rough edges IMHO. In theory, it allows a much more coherent packaging, but in practice, you still need to integrate it into your mail and web server environment. Sharing files as a communication mechanism is fragile, and for example getting the postfix maps generated by Mailman into your postfix installation which may be in another container, on the docker host, or on another server, can be tricky.
Cheers, Hans-Martin
participants (6)
-
Abhilash Raj -
Hans-Martin Mosner -
ieso＠johnwillson.com -
Jonathan M -
Mark Sapiro -
Stephen J. Turnbull