share directories between postfix and mailman 3
Hi there,
I think I have (finally) set up a working instance of mailman 3 on a host separate from the companion postfix mail server actually sending out the mails. One server (postfix) is directly in the internet and the other one is in an internal network not directly reachable from the internet (but connected using a reverse proxy). The services on the hosts are available to one another (i. e. the postfix server is reachable using 127.0.0.1:25 from mailman and mailman is reachable using 127.0.0.1:8024 from postfix using a permanent ssh tunnel betwenn the two hosts). I think that this solution could work quite reliably... (at least I hope so.)
But that's not the full deal. Postfix needs access to some hashed configuration files containing the transport maps and the domains of the mailman installation, and mailman needs the contents of the postfix configuration file. As a quick and dirty solution, I've established a process which periodically copies the /var/lib/mailman3 directory from the mailman box to the postfix box using rsync. But I would like to have a more professional solution, like a directly shared directory between both. I did not yet find any suggestions on how to achieve this. Are there any best practices recommendations? I already tried sharing the directories using sshfs, but that did not work.
I connected the remote directory postfix.example.com:/var/lib/mailman3 to my local directory /var/lib/mailman3 using
sshfs root@postfix.example.com:/var/lib/mailman3 /var/lib/mailman3
This seemed to work pretty fine, as all files and directories seemed to be available, but when starting mailman 3, the following error message was shown: FileExistsError: A race condition might have happened. /var/lib/mailman3 actually exists and is not a directory.
Any idea why this message is shown? So, is it a bad idea to use sshfs?
Do I have to install an NFS server? But I am not sure if sharing directories using NFS across the Internet is a good idea... Any suggestions?
Best regards Tom
On 1/22/24 13:05, Thomas Schachtner via Mailman-users wrote:
Hi there,
But that's not the full deal. Postfix needs access to some hashed configuration files containing the transport maps and the domains of the mailman installation, and mailman needs the contents of the postfix configuration file.
Why do you think Mailman needs the contents of the postfix configuration file.. I don't think so.
As a quick and dirty solution, I've established a process which periodically copies the /var/lib/mailman3 directory from the mailman box to the postfix box using rsync.
It appears you are using the Debian/Ubuntu package or at least have
configured layout: fhs in the [mailman] section of mailman.cfg. This
is OK, but in any case, all Postfix needs is /var/lib/mailman3/data/*.
But I would like to have a more professional solution, like a directly shared directory between both. I did not yet find any suggestions on how to achieve this. Are there any best practices recommendations? I already tried sharing the directories using sshfs, but that did not work.
I connected the remote directory postfix.example.com:/var/lib/mailman3 to my local directory /var/lib/mailman3 using
sshfs root@postfix.example.com:/var/lib/mailman3 /var/lib/mailman3
I would suggest the opposite. I.e. on the postfix machine
sshfs user@mailman_machine:/var/lib/mailman3/data /var/lib/mailman3/data
This seemed to work pretty fine, as all files and directories seemed to be available, but when starting mailman 3, the following error message was shown: FileExistsError: A race condition might have happened. /var/lib/mailman3 actually exists and is not a directory.
Any idea why this message is shown?
Possibly because the uid and gid of /var/lib/mailman3 on the postfix machine do not map to the appropriate Mailman uid and gid on the mailman machine.
So, is it a bad idea to use sshfs?
If you do it the other way, it might work.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
On 1/22/24 13:05, Thomas Schachtner via Mailman-users wrote:
Hi there,
But that's not the full deal. Postfix needs access to some hashed configuration files containing the transport maps and the domains of the mailman installation, and mailman needs the contents of the postfix configuration file.
Why do you think Mailman needs the contents of the postfix configuration file.. I don't think so. I thought so, as mailman does not start if there's not /etc/postfix/main.cf file is there.
The following messages are logged: Jan 23 21:20:35 mailman mailman3[491404]: RuntimeError: command failure: /usr/sbin/postmap /var/lib/mailman3/data/postfix_lmtp, 1, Operation not permitted Jan 23 21:20:35 mailman mailman3[491404]: command failure: /usr/sbin/postmap /var/lib/mailman3/data/postfix_domains, 1, Operation not permitted
If /etc/postfix/main.cf is there, mailman is starting perfectly fine. But in another anwer I learned that mailman might not need the contents of that file but rather change it (although I did not see any changes).
As a quick and dirty solution, I've established a process which periodically copies the /var/lib/mailman3 directory from the mailman box to the postfix box using rsync.
It appears you are using the Debian/Ubuntu package or at least have configured
layout: fhsin the[mailman]section of mailman.cfg. This is OK, but in any case, all Postfix needs is /var/lib/mailman3/data/*.
Thanks for this information. I already thought so, but I was not sure. Yes, I am using the Debian package. I also found the line with 'layout', but it reads 'layout: debian'. They write: "You should not change this variable"
But I would like to have a more professional solution, like a directly shared directory between both. I did not yet find any suggestions on how to achieve this. Are there any best practices recommendations? I already tried sharing the directories using sshfs, but that did not work.
I connected the remote directory postfix.example.com:/var/lib/mailman3 to my local directory /var/lib/mailman3 using
sshfs root@postfix.example.com:/var/lib/mailman3 /var/lib/mailman3
I would suggest the opposite. I.e. on the postfix machine
sshfs user@mailman_machine:/var/lib/mailman3/data /var/lib/mailman3/data
There are two reasons why I would like to do it the other way around: (1) I would like to make all changes on the mailman3 box and change as little as possible on the postfix server and (2) mailman is running on a system which does not have a fixed IP address. Mounting the file system from there can become complicated, as the IP address can change on a daily basis...
This seemed to work pretty fine, as all files and directories seemed to be available, but when starting mailman 3, the following error message was shown: FileExistsError: A race condition might have happened. /var/lib/mailman3 actually exists and is not a directory.
Any idea why this message is shown?
Possibly because the uid and gid of /var/lib/mailman3 on the postfix machine do not map to the appropriate Mailman uid and gid on the mailman machine. You refer to the uid and gid of the "list" user? They are both 38 (uid and gid) on both machines.
So, is it a bad idea to use sshfs?
If you do it the other way, it might work.
I try to work around the changing IP address challenge then. Thanks! Tom
On 1/23/24 12:30, Thomas Schachtner via Mailman-users wrote:
Mark Sapiro wrote:
Why do you think Mailman needs the contents of the postfix configuration file.. I don't think so.
I thought so, as mailman does not start if there's not /etc/postfix/main.cf file is there.
The following messages are logged: Jan 23 21:20:35 mailman mailman3[491404]: RuntimeError: command failure: /usr/sbin/postmap /var/lib/mailman3/data/postfix_lmtp, 1, Operation not permitted Jan 23 21:20:35 mailman mailman3[491404]: command failure: /usr/sbin/postmap /var/lib/mailman3/data/postfix_domains, 1, Operation not permitted
Mailman itself is not looking for Postfix main.cf, but when Pustfix is the MTA, it invokes the postmap command to create the .db files for postfix_lmtp and postfix_domains. Apparently postmap fails if there is not a Postfix main.cf.
You can avoid this by using regexp tables instead of hash tables. See <https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...>.
This seemed to work pretty fine, as all files and directories seemed to be available, but when starting mailman 3, the following error message was shown: FileExistsError: A race condition might have happened. /var/lib/mailman3 actually exists and is not a directory.
Any idea why this message is shown?
Possibly because the uid and gid of /var/lib/mailman3 on the postfix machine do not map to the appropriate Mailman uid and gid on the mailman machine. You refer to the uid and gid of the "list" user? They are both 38 (uid and gid) on both machines.
Then that isn't the reason. I don't know then why Mailman would complain that /var/lib/mailman3 actually exists and is not a directory. This message comes from mailman.utilities.filesystem.makedirs(). The code is at <https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/utilities/files...>.
makedirs() calls first_inexistent_directory() which checks if the directory exists with os.path.isdir() which apparently says it isn't a directory which makes no sense unless there is an actual race condition and you are creating the /var/lib/mailman3 directory on the mailman machine at the same time as you are starting mailman.
BTW, you say you are doing
sshfs root@postfix.example.com:/var/lib/mailman3 /var/lib/mailman3
on the mailman machine. Presumably on the mailman machine /var/lib/mailman3 has lots of content. How does that work?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thomas Schachtner via Mailman-users writes:
But that's not the full deal. Postfix needs access to some hashed configuration files containing the transport maps and the domains of the mailman installation,
The transport is constant: lmtp:[$IP_OF_MAILMAN]:$LMTP_PORT. What the MTA needs is to confirm domains and list addresses. Individual addresses can be confirmed by your SQL RDBMS or by Mailman over TCP/IP. The Postgres implementation for Postfix is here:
https://gitlab.com/mailman/mailman/-/merge_requests/1161
It's not ready for merge yet, but it has been used for 6 months in production on a site with ~20k lists and ~100k posts/day. I think the same approach works for MySQL and SQLite3. The callout implementation for Exim is here:
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...
This probably can be adapted to Postfix but I haven't tried. See https://www.postfix.org/ADDRESS_VERIFICATION_README.html
Note that one of the benefits of MR #1161 is that for large sites it dramatically speeds up the process of creating lists by inhibiting the creation of the Postfix transport map files. It's not necessary for either approach, but the difference is noticable at O(1000) lists, and at 10000 lists a quite powerful host (with Postgres on a separate host) took 24 hours merely to migrate list configurations (and it appears the time cost is quadratic, so we gave up and devised the "ask Postgres" approach). Mailman cold start was taking ~15 minutes, because it was recreating that file. You'd get the same benefits of that patch when using the callout approach.
and mailman needs the contents of the postfix configuration file.
Backwards. Mailman *creates* the Postfix configuration file but otherwise does not use it.
Are there any best practices recommendations [for sharing file systems]?
Docker clusters usually use NFS, I think. That's what the AWS configs I've heard of use, too (AWS-branded, of course, but under the hood it's NFS). If you're worried about exposing NFS to the Internet of Threats, firewalls permitting only the Mailman and Postfix hosts to access the NFS service, and certificate-authenticated TLS for the connections should be as safe as anything exposed to the Internet.
The minimal attack surface approach is the LMTP RCPT TO callout approach in the Exim configuration above. Mailman's LMTP service must be accessible to the MTA anyway, so then you don't need to open any other ports such as Postgres or NFS.
Steve
Hi Steve,
thanks for your explanations!
[...] What the MTA needs is to confirm domains and list addresses. Individual addresses can be confirmed by your SQL RDBMS or by Mailman over TCP/IP. The Postgres implementation for Postfix is here:
https://gitlab.com/mailman/mailman/-/merge_requests/1161
It's not ready for merge yet, but it has been used for 6 months in production on a site with ~20k lists and ~100k posts/day. I think the same approach works for MySQL and SQLite3. The callout implementation for Exim is here:
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...
This probably can be adapted to Postfix but I haven't tried. See https://www.postfix.org/ADDRESS_VERIFICATION_README.html
and mailman needs the contents of the postfix configuration file.
Backwards. Mailman *creates* the Postfix configuration file but otherwise does not use it. I just saw that mailman does not start (systemctl start mailman3.service) on my debian installation if the postfix configuration file is not there, so I thought it might be needed by mailman3. But maybe this is not a mailman issue but a debian-only issue... Nevertheless, I just put the file on the mailman box and everything is fine. It does not seem to be changed when starting mailman, so maybe
I'll have a look at the database approach you mentioned: that's safe...
Are there any best practices recommendations [for sharing file systems]?
Docker clusters usually use NFS, I think. That's what the AWS configs I've heard of use, too (AWS-branded, of course, but under the hood it's NFS). If you're worried about exposing NFS to the Internet of Threats, firewalls permitting only the Mailman and Postfix hosts to access the NFS service, and certificate-authenticated TLS for the connections should be as safe as anything exposed to the Internet. One of the two systems (mailman) is currently running on a dial-up line with regularly changing IP addresses, so a static NFS mount might be difficult to establish. The minimal attack surface approach is the LMTP RCPT TO callout approach in the Exim configuration above. Mailman's LMTP service must be accessible to the MTA anyway, so then you don't need to open any other ports such as Postgres or NFS. I'll definitely have a look at the callout approach (currently, I have no clue what that might be...), but I also see that exposing LMTP to the Internet is dangerous...
Thank you! Tom
Thomas Schachtner via Mailman-users writes:
I just saw that mailman does not start (systemctl start mailman3.service) on my debian installation if the postfix configuration file is not there, so I thought it might be needed by mailman3.
Mark explained this.
But maybe [the Mailman won't start issue] is not a mailman issue but a debian-only issue...
It's that Mailman 3 recreates the postmapped database files at startup. It will happen on any stock Mailman 3 + Postfix installation if postmap can't find main.cf.
Nevertheless, I just put the file on the mailman box and everything is fine. It does not seem to be changed when starting mailman, so maybe that's safe...
At startup, Mailman reads the postfix-lmtp and postfix domains files, and (re-)creates the .db files using Postfix's postmap utility. When creating a domain or list, it adds an entry to the appropriate file, and then compiles it with postmap. Otherwise Mailman does not touch or even read those files.
I'll definitely have a look at the callout approach (currently, I have no clue what that might be...), but I also see that exposing LMTP to the Internet is dangerous...
Exposing anything to the Internet is dangerous.
Since LMTP is (usually) cleartext and unauthenticated (I'm pretty sure both Postfix and Mailman can do authenticated TLS), yes, it's riskier than say ssh. That's why I suggest firewalls and TLS connections, and then you're basically as secure as ssh. Or you can avoid exposing LMTP by having Postfix on the Mailman host, and use virtual alias or mailbox hosts with virtual_transport = lmtp:[127.0.0.1]:8024 to catch the Mailman traffic.
Steve
participants (3)
-
Mark Sapiro -
Stephen J. Turnbull -
Thomas Schachtner