ieso@johnwillson.com wrote:
@AndrewH.
Yes, I saw your notes, but the impression I got during this effort is that package-based/non-docker implementations change too much between versions, so anything written based on Ubuntu 16.04 or earlier just didn't feel worth spending significant time on... that impression came from the "this is new to version >18.04" warnings in the Ubuntu 18.04 walkthrough ( https://docs.google.com/document/d/1xIcSsoNFp2nHi7r4eQys00s9a0k2sHhu1V5Plant...). See my comment below for more.
Upgrading for me is just a case of running the relevant Pip commands and running the post-update script. I have done it a number of times. As I have done the installation from scratch I am fairly confident I can move to a newer OS without much hassle and identify where problems may occur. The biggest issue for me is getting modules via Pip and having issues therein. For example when I first did this in March I ran into an issue with a Python module having been upgraded and thus the latest version of a package caused an error which took a while to troubleshoot until I wrote to this list and eventually installed an older version.
@JonathanM
My attempt at doing a DigitalOcean package/non-docker based implementation was frustrating. I found a really detailed walkthrough ( https://docs.google.com/document/d/1xIcSsoNFp2nHi7r4eQys00s9a0k2sHhu1V5Plant...), and after the usual hyperkitty authorization problem solving, it got me to a point where everything looked perfect, based on a glance at the web interface. However, a bizarre collection of errors showed up during QA testing, such as the following:
I think the best point is to give us errors if you encounter them and we can try and troubleshoot specifics rather than giving up altogether, however, see below on the Docker part.
So here's my latest (failed) attempt, to see if anyone can spot where I went wrong. Again, I'm just trying to get a dedicated mailman3 server up from scratch. Oh, I should add that I tried the mailman4 AWS Mailman3-as-SaaS <https://aws.amazon.com/marketplace/pp/New-IT-Mailman-4/B089M2PCVV>, and that >worked just fine. However we can't host our data on a SaaS platform, so that's unfortunately not an option for us. I just want to mention it for anyone looking for a "give me mailman services in five minutes for $30/month" solution.
Regarding this marketplace image this would actually run on your VM but its running as an appliance/turnkey solution so may still not qualify for you. I wouldn't go there anyway because troubleshooting it may be hard and I am a bit suspicious of smaller companies posting to Amazon Marketplace like this as to their long-term support.
Phase 1 - Prepare the AWS instance and MTA
Most of the documentation on the Mailman side talks about running these containers on a dedicated VM with an MTA and front-end web server installed on the VM. If you are using a VM to run the containers then the network 172.19.199.0/24 is assumed to be a backend Docker network which is used to allow the different containers to talk to each other and to allow the front-end web server/MTA to communicate. If it were me doing this with this setup I wouldn't create the VPC with the same subnet as it may cause issues and we don't really care of the VPC subnet details. If you look at the Docker Compose file in the Docker-Mailman repo you can see its creating this 172.19.199.0/24 network as a bridge network and in most cases you wouldn't connect to these containers from anything other than the VM.
A typical traffic flow may look something like the following:
- Inbound SMTP to Exim/Postfix running on the VM (inbound access to port 25 and MX delivery arranged).
- Exim/Postfix knows to relay messages for the Mailman domains to 172.19.199.2 on port 8024 (LMTP).
- Mailman services connects to Postgresql DB on 172.19.199.4.
- Mailman Hyperkitty plugin connects to the Hyperkitty web service on 172.19.199.3:8000 to archive the message.
- Mailman core sends email out to Exim/Postfix server running on 172.19.199.1. What Postfix does with the email (i.e sends out via SES) is not Mailman's concern.
- A user wishing to access the web interface connects to the VM directly on port 80 or preferably 443 with necessary inbound NSG rules defined.
- The front-end web server knows to proxy these requests to 172.19.199.3:8000 (or if the front-end server supports Uwsgi then its 172.19.199.3:8080
- The front-end web server knows that the /static path should be served up locally from the local filesystem /opt/mailman/web/static which the Mailman-web container writes files to when it starts up.
I think its best to continue running these containers from the VM in the above setup, you mention ECS later on in this email but that is going to confuse the issue unless you really want to run the containers under ECS.
[...]
- Tried to follow the standard directions... .I put most of the relevant files below.
Did you create the necessary /opt/mailman/core and /opt/mailman/web directories on the VM and set perms as detailed in the repo README?
docker-compose.yaml
[...]
services: mailman-core: [...] volumes:
- /opt/mailman/core:/opt/mailman/
/opt/mailman/core (or at least /opt/mailman) on the host VM needs to exist. The container will write files to /opt/mailman which then get persistently written to /opt/mailman/core on the host. One of the files that get written in this way is /opt/mailman/core/var/data/postfix_lmtp and /opt/mailman/core/var/data/postfix_domains which is referred to in your Postfix configuration (see later on).
[...]
maillog has some interesting stuff I'm seeing for the first time. Let me know if this suggests any obvious suspects.
[...]
postfix/qmgr[4351]: error: open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Check this file on the VM and see if it exists (or if parent path exists). If parent /opt/mailman/core doesn't exist create it. If it exists but there is no var directory inside it, check the container stdout logs for errors.
If other directories inside /opt/mailman/core/var (especially logs) look at them for more info.
Thanks. Andrew.