External MTA incoming mail: configuration
I have mailman set up to send mail via an external MTA, which is supported by mailman and described in the documentation here: https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht....
Specifically it says: |lmtp_host|and|lmtp_port|are parameters which are used by Mailman, but also will be passed to the MTA to identify the Mailman host. The “same host” case is special; some MTAs (including Postfix) do not recognize “localhost”, and need the numerical IP address. If they are on different hosts,|lmtp_host|should be set to the domain name or IP address of the Mailman host.|lmtp_port|is fairly arbitrary (there is no standard port for LMTP). Use any port convenient for your site. “8024” is as good as any, unless another service is using it.
However, there is a practical problem with LMTP (to receive incoming email) which is as follows:
The setup in mailman.cfg:
lmtp_host: lists.fast.za.net lmtp_port: 8024
Now, lists.fast.za.net is the mailman server. The docs say that mailman passes these on to the MTA. However, the MTA (postfix) on box2.gtahardware.co.za has the following setting in it main.cf:
virtual_transport=lmtp:[127.0.0.1]:10025
If I reply to a signup confirmation email for example, the mail is received and processed by box2.gtahardware.co.za, and it is not passed on to the mailman server.
So what is meant by "but also will be passed to the MTA to identify the Mailman host" in the docs?
If I change the config of the MTA (postfix) to be "virtual_transport=lmtp:[lists.fast.za.net]:8024", I assume the incoming mail will be forwarded, but how does postfix know that only the mailman emails should be forwarded and not all the other mail that the server at box2.gtahardware.co.za receives?
This is quite fuzzt to me at this stage. Could someone help me with some clarity please?
On 7/30/24 11:18, Roland Giesler via Mailman-users wrote:
Now, lists.fast.za.net is the mailman server. The docs say that mailman passes these on to the MTA. However, the MTA (postfix) on box2.gtahardware.co.za has the following setting in it main.cf:
The docs are badly worded. "but also will be passed to the MTA to identify the Mailman host." means passed manually in the MTA's configuration.
virtual_transport=lmtp:[127.0.0.1]:10025
You should not be using virtual_transport. You should be using transport_maps. See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht...
If I reply to a signup confirmation email for example, the mail is received and processed by box2.gtahardware.co.za, and it is not passed on to the mailman server.
It appears you have a situation where the MX for the Mailman domains is
not the Mailman server. You need to configure the MTA on
box2.gtahardware.co.za to relay Mailman's mail to the mailman server.
You can to this by configuring that MTA to use transport_maps, etc as if
it were the Mailman server. In order to get Mailman's generated files on
the other server, you need to create a process to do that. When Mailman
makes changes by adding or deleting lists or when it is started, it
updates the postfix_lmtp and postfix_domains files and calls a command
(normally postmap) to update the corresponding .db files. The command
can be set by setting postmap_command in the [postfix] section in
mailman.cfg.
You can set this to a process which accepts a file name argument and then runs postmap on the file and rsyncs the file and it's .db to the other server.
So what is meant by "but also will be passed to the MTA to identify the Mailman host" in the docs?
As I said, that's badly worded it means 'passed manually to the MTA's config'.
If I change the config of the MTA (postfix) to be "virtual_transport=lmtp:[lists.fast.za.net]:8024", I assume the incoming mail will be forwarded, but how does postfix know that only the mailman emails should be forwarded and not all the other mail that the server at box2.gtahardware.co.za receives?
Using transport_maps you can forward only list mail.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 7/30/24 13:39, Mark Sapiro wrote:
The docs are badly worded. "but also will be passed to the MTA to identify the Mailman host." means passed manually in the MTA's configuration.
I have updated the doc. it now says "lmtp_host and lmtp_port are parameters which are used by Mailman, but also need to be configured in the MTA to identify the Mailman host."
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024/07/30 22:39, Mark Sapiro wrote:
It appears you have a situation where the MX for the Mailman domains is not the Mailman server. You need to configure the MTA on box2.gtahardware.co.za to relay Mailman's mail to the mailman server. You can to this by configuring that MTA to use transport_maps, etc as if it were the Mailman server. In order to get Mailman's generated files on the other server, you need to create a process to do that. When Mailman makes changes by adding or deleting lists or when it is started, it updates the postfix_lmtp and postfix_domains files and calls a command (normally postmap) to update the corresponding .db files. The command can be set by setting
postmap_commandin the[postfix]section in mailman.cfg.
I'm not able to find any documentation about postmap_command, not even
in the schema.cfg. If I have to provide method to update the
postfix_lmtp and postfix_domains on another server, then it would have
to something like sftp? mailman changes the files on the mailman server
(in a location I set) and then postmap_command copies those files to the
destination on the MTA server. Is that in principle what needs to be done?
thanks again!
Roland
On Wed, Jul 31, 2024 at 2:35 PM Roland Giesler via Mailman-users < mailman-users@mailman3.org> wrote:
On 2024/07/30 22:39, Mark Sapiro wrote:
It appears you have a situation where the MX for the Mailman domains is not the Mailman server. You need to configure the MTA on box2.gtahardware.co.za to relay Mailman's mail to the mailman server. You can to this by configuring that MTA to use transport_maps, etc as if it were the Mailman server. In order to get Mailman's generated files on the other server, you need to create a process to do that. When Mailman makes changes by adding or deleting lists or when it is started, it updates the postfix_lmtp and postfix_domains files and calls a command (normally postmap) to update the corresponding .db files. The command can be set by setting
postmap_commandin the[postfix]section in mailman.cfg.I'm not able to find any documentation about
postmap_command, not even in the schema.cfg. If I have to provide method to update the postfix_lmtp and postfix_domains on another server, then it would have to something like sftp? mailman changes the files on the mailman server (in a location I set) and then postmap_command copies those files to the destination on the MTA server. Is that in principle what needs to be done?thanks again!
Read this thread and take a keen look at the solution provided by @Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> :
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/E...
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 2024/07/31 13:47, Odhiambo Washington via Mailman-users wrote:
On Wed, Jul 31, 2024 at 2:35 PM Roland Giesler via Mailman-users < mailman-users@mailman3.org> wrote:
On 2024/07/30 22:39, Mark Sapiro wrote:
It appears you have a situation where the MX for the Mailman domains is not the Mailman server. You need to configure the MTA on box2.gtahardware.co.za to relay Mailman's mail to the mailman server. You can to this by configuring that MTA to use transport_maps, etc as if it were the Mailman server. In order to get Mailman's generated files on the other server, you need to create a process to do that. When Mailman makes changes by adding or deleting lists or when it is started, it updates the postfix_lmtp and postfix_domains files and calls a command (normally postmap) to update the corresponding .db files. The command can be set by setting
postmap_commandin the[postfix]section in mailman.cfg.I'm not able to find any documentation about
postmap_command, not even in the schema.cfg. If I have to provide method to update the postfix_lmtp and postfix_domains on another server, then it would have to something like sftp? mailman changes the files on the mailman server (in a location I set) and then postmap_command copies those files to the destination on the MTA server. Is that in principle what needs to be done?Read this thread and take a keen look at the solution provided by @Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> :
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/E...
Excellent! I think I can get the sshfs route working, but the pending database solution is elegant! I love it! Thanks for the pointer.
On 2024/07/31 13:47, Odhiambo Washington via Mailman-users wrote:
Read this thread and take a keen look at the solution provided by @Stephen J. Turnbull<turnbull.stephen.fw@u.tsukuba.ac.jp> :
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/E...
The reason why he got the error is indeed because of fs permissions. I added a line into usr/lib/python3/dist-packages/mailman/utilities/filesystem.py at line 69 to print the directory and it said:
Jul 31 18:02:01 mailman mailman3[1029]: ls: cannot access '/var/lib/mailman3': Permission denied
So I changed the permissions to the user that mailman runs under (on both servers uid 38).
sshfs -o uid=38,gid=38 -o allow_other root@box2.gtahardware.co.za:/etc/mailman3/data /var/lib/mailman3/data
Now mailman3 starts without errors and I have a remote directory into which the domains can be written by mailman3's django admin interface.
I needed to put these files at these locations:
transport_maps = hash:/etc/mailman3/data/postfix_lmtp local_recipient_maps = hash:/etc/mailman3/data/postfix_lmtp relay_domains = hash:/etc/mailman3/data/postfix_domains
However, since I'm using power-mailinabox, Postfix is configured to use sqlite already and has these lines by default:
virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf virtual_mailbox_maps=sqlite:/etc/postfix/virtual-mailbox-maps.cf virtual_alias_maps=sqlite:/etc/postfix/virtual-alias-maps.cf local_recipient_maps=$virtual_mailbox_maps
So I have only added this line to postfix's main.cf:
# Add postfix_domains path so mailman can add domains to it for lists relay_domains=hash://etc/mailman3/data/postfix_domains
The local_recepient_maps are set my power-mailinabox already, so I'm ignoring what mailman creates.
However, in the process I have also change the list domain to be different from the MTA's. Postfix accepts mail for fast.za.net, so I created lists.fast.za.net for the list server. I have changed the domain in django to reflect this, but now I get the original error again, since mailman-web tries to authenticate as lists.fast.za.net, which the MTA doesn't know. I'll have to dig a little to see why this is.
On 2024/08/01 10:09, Roland Giesler via Mailman-users wrote:
I needed to put these files at these locations:
transport_maps = hash:/etc/mailman3/data/postfix_lmtp local_recipient_maps = hash:/etc/mailman3/data/postfix_lmtp relay_domains = hash:/etc/mailman3/data/postfix_domains
However, since I'm using power-mailinabox, Postfix is configured to use sqlite already and has these lines by default:
virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf virtual_mailbox_maps=sqlite:/etc/postfix/virtual-mailbox-maps.cf virtual_alias_maps=sqlite:/etc/postfix/virtual-alias-maps.cf local_recipient_maps=$virtual_mailbox_maps
So I have only added this line to postfix's main.cf:
# Add postfix_domains path so mailman can add domains to it for lists relay_domains=hash://etc/mailman3/data/postfix_domains
The local_recipient_maps are set by power-mailinabox already, so I'm ignoring what mailman creates.
It seems that this is a mistake. If I don't let postfix read the local_recipient_maps that mailman3 created, the incoming emails are not sent to the other host. So I have a conflict with the local_recipient_maps that postfix has (created by Power-mailinabox) and those created by mailman3.
I'm hoping that if I specify local_recipient_maps more than once and then both sources will be considered. Would that work?
If not, then I'm pretty much stumped, unless I go for the database option that has not been merged yet. Even if I do that, I would need to make sure that mailman3 doesn't remove the other entries in the database which, judging by what I've seen, seems to be the case.
Any ideas?
On 2024/08/02 16:50, Roland Giesler via Mailman-users wrote:
I'm hoping that if I specify local_recipient_maps more than once and then both sources will be considered. Would that work?
It turns out that I can specify multiple comma separated options on the local_recipient_maps = line, so that's great!
It seems though that since I'm not running a local MTA, when then mail arrives at the mailman3 server, there's nothing that can process it. So should I run postfix in it's most basic setup to allow mail to be received or what is the best way to deal with it? Mailman3 has to receive mail like signup confirmations, etc after all, not?
On Fri, Aug 2, 2024 at 6:12 PM Roland Giesler via Mailman-users < mailman-users@mailman3.org> wrote:
On 2024/08/02 16:50, Roland Giesler via Mailman-users wrote:
I'm hoping that if I specify local_recipient_maps more than once and then both sources will be considered. Would that work?
It turns out that I can specify multiple comma separated options on the local_recipient_maps = line, so that's great!
It seems though that since I'm not running a local MTA, when then mail arrives at the mailman3 server, there's nothing that can process it. So should I run postfix in it's most basic setup to allow mail to be received or what is the best way to deal with it? Mailman3 has to receive mail like signup confirmations, etc after all, not?
MM3 has to receive mail, yes. If it doesn't, then it sits there processing nothing for the MLs.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/2/24 08:12, Roland Giesler via Mailman-users wrote:
It seems though that since I'm not running a local MTA, when then mail arrives at the mailman3 server, there's nothing that can process it. So should I run postfix in it's most basic setup to allow mail to be received or what is the best way to deal with it? Mailman3 has to receive mail like signup confirmations, etc after all, not?
If you use transport_maps on the remote MTA to relay the list mail to port 8024 on the Mailman server, Mailman's LMTP runner will receive the mail and you don't need a local MTA.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Fri, Aug 2, 2024 at 6:44 PM Mark Sapiro <mark@msapiro.net> wrote:
On 8/2/24 08:12, Roland Giesler via Mailman-users wrote:
It seems though that since I'm not running a local MTA, when then mail arrives at the mailman3 server, there's nothing that can process it. So should I run postfix in it's most basic setup to allow mail to be received or what is the best way to deal with it? Mailman3 has to receive mail like signup confirmations, etc after all, not?
If you use transport_maps on the remote MTA to relay the list mail to port 8024 on the Mailman server, Mailman's LMTP runner will receive the mail and you don't need a local MTA.
Nice one! I'd love to know how this config is done in Postfix. Heading to ask Uncle G.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 8/2/24 08:47, Odhiambo Washington via Mailman-users wrote:
Nice one! I'd love to know how this config is done in Postfix.
Just like https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht... or https://docs.mailman3.org/en/latest/install/virtualenv.html#setup-mta only on the remote MTA.
This requires copying the var/data/postfix* files from the Mailman server to the remote MTA server which can be automated as mentioned earlier in this thread.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024/08/02 17:43, Mark Sapiro wrote:
On 8/2/24 08:12, Roland Giesler via Mailman-users wrote:
It seems though that since I'm not running a local MTA, when then mail arrives at the mailman3 server, there's nothing that can process it. So should I run postfix in it's most basic setup to allow mail to be received or what is the best way to deal with it? Mailman3 has to receive mail like signup confirmations, etc after all, not?
If you use transport_maps on the remote MTA to relay the list mail to port 8024 on the Mailman server, Mailman's LMTP runner will receive the mail and you don't need a local MTA.
Ah, so I just have to figure out why the LMTP runner is not running.
mailman:/var/lib/mailman3# ps ax PID TTY STAT TIME COMMAND 1 ? Ss 0:07 /sbin/init 42 ? Ss 0:07 /lib/systemd/systemd-journald 90 ? Ss 0:00 /lib/systemd/systemd-networkd 98 ? Ss 0:00 /lib/systemd/systemd-resolved 101 ? Ss 0:01 /usr/sbin/cron -f -P 102 ? Ss 0:00 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 106 ? Ss 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers 107 ? Ssl 0:01 /usr/sbin/rsyslogd -n -iNONE 108 ? Ss 0:00 /lib/systemd/systemd-logind 111 ? Ss 0:07 /usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini 115 pts/0 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux 116 pts/1 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty1 115200,38400,9600 linux 117 pts/2 Ss+ 0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud tty2 115200,38400,9600 linux 127 ? Ss 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; 129 ? S 0:00 nginx: worker process 138 ? Ssl 28:01 /usr/sbin/mysqld 277 ? Sl 0:05 /usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini 278 ? Ss 0:00 /bin/sh -c python3 manage.py qcluster 282 ? S 0:01 python3 manage.py qcluster 328 ? Ss 0:00 /usr/lib/postfix/sbin/master -w 331 ? S 0:00 qmgr -l -t unix -u 340 ? S 3:49 python3 manage.py qcluster 341 ? S 0:00 python3 manage.py qcluster 342 ? S 0:00 python3 manage.py qcluster 343 ? S 0:00 python3 manage.py qcluster 344 ? S 0:00 python3 manage.py qcluster 345 ? S 0:00 python3 manage.py qcluster 346 ? S 0:00 python3 manage.py qcluster 347 ? S 0:00 python3 manage.py qcluster 348 ? S 0:00 python3 manage.py qcluster 349 ? S 0:00 python3 manage.py qcluster 350 ? S 24:13 python3 manage.py qcluster 354 ? Ss 0:00 /lib/systemd/systemd --user 355 ? S 0:00 (sd-pam) 953 ? Ss 0:00 sshd: root@pts/3 972 pts/3 Ss 0:00 -bash 1284 pts/3 S 0:00 ssh -x -a -oClearAllForwardings=yes -2 root@box2.gtahardware.co.za -s sftp 1285 ? Ssl 0:00 sshfs -o uid=38,gid=38 -o allow_other root@box2.gtahardware.co.za:/etc/mailman3/data /var/lib/mailman3/data 1302 ? Ss 0:00 /usr/bin/python3 /usr/lib/mailman3/bin/master --force -C /etc/mailman3/mailman.cfg 1308 ? S 0:18 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=archive:0:1 1309 ? S 0:07 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=bounces:0:1 1310 ? S 0:17 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=command:0:1 1311 ? S 0:16 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=in:0:1 1313 ? S 0:24 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=nntp:0:1 1314 ? S 0:16 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=out:0:1 1315 ? S 0:18 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=pipeline:0:1 1316 ? S 0:18 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 1317 ? S 0:01 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=retry:0:1 1318 ? S 0:02 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=task:0:1 1319 ? S 0:18 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=virgin:0:1 1320 ? S 0:17 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=digest:0:1 13459 ? S 0:01 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 13460 ? S 0:01 /usr/bin/python3 /usr/lib/mailman3/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 20765 ? S 0:00 pickup -l -t unix -u -c 21109 pts/3 R+ 0:00 ps ax
mailman:/var/lib/mailman3# netstat -tunap Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 98/systemd-resolved tcp 0 0 127.0.0.1:33060 0.0.0.0:* LISTEN 138/mysqld tcp 0 0 127.0.0.1:8001 0.0.0.0:* LISTEN 1316/python3 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 127/nginx: master p tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 127/nginx: master p tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 328/master tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 138/mysqld tcp 0 0 127.0.0.1:56588 127.0.0.1:3306 ESTABLISHED 13459/python3 tcp 0 0 127.0.0.1:3306 127.0.0.1:50542 ESTABLISHED 138/mysqld tcp 0 0 127.0.0.1:3306 127.0.0.1:50530 ESTABLISHED 138/mysqld tcp 0 0 192.168.161.103:60010 197.214.119.180:22 ESTABLISHED 1284/ssh tcp 0 0 127.0.0.1:3306 127.0.0.1:59222 ESTABLISHED 138/mysqld tcp 0 0 127.0.0.1:3306 127.0.0.1:46810 ESTABLISHED 138/mysqld tcp 0 0 127.0.0.1:50530 127.0.0.1:3306 ESTABLISHED 1319/python3 tcp 0 0 127.0.0.1:46796 127.0.0.1:3306 ESTABLISHED 1318/python3 tcp 0 0 127.0.0.1:46810 127.0.0.1:3306 ESTABLISHED 1309/python3 tcp 0 0 127.0.0.1:59222 127.0.0.1:3306 ESTABLISHED 13460/python3 tcp 0 0 127.0.0.1:50542 127.0.0.1:3306 ESTABLISHED 1314/python3 tcp 0 0 127.0.0.1:3306 127.0.0.1:46796 ESTABLISHED 138/mysqld tcp 0 0 127.0.0.1:3306 127.0.0.1:56588 ESTABLISHED 138/mysqld tcp6 0 0 ::1:25 :::* LISTEN 328/master tcp6 0 0 :::443 :::* LISTEN 127/nginx: master p tcp6 0 0 :::22 :::* LISTEN 1/init tcp6 0 0 :::80 :::* LISTEN 127/nginx: master p tcp6 0 0 192.168.161.103:22 192.168.131.150:43924 ESTABLISHED 1/init udp 0 0 127.0.0.53:53 0.0.0.0:* 98/systemd-resolved
and I have in mailman.cfg:
mailman:/var/lib/mailman3# cat /etc/mailman3/mailman.cfg | grep "^[^#;]" [mailman] site_owner: mailman@fast.za.net noreply_address: noreply default_language: en sender_headers: from from_ reply-to sender email_commands_max_lines: 10 pending_request_life: 3d cache_life: 7d pre_hook: post_hook: layout: debian filtered_messages_are_preservable: no html_to_plain_text_command: /usr/bin/lynx -dump $filename listname_chars: [-_.0-9a-z] [shell] prompt: >>> banner: Welcome to the GNU Mailman shell use_ipython: no history_file: [paths.debian] var_dir: /var/lib/mailman3 queue_dir: $var_dir/queue bin_dir: /usr/lib/mailman3/bin list_data_dir: $var_dir/lists log_dir: /var/log/mailman3 lock_dir: $var_dir/locks data_dir: $var_dir/data cache_dir: $var_dir/cache etc_dir: /etc/mailman3 ext_dir: $var_dir/ext messages_dir: $var_dir/messages archive_dir: $var_dir/archives template_dir: $var_dir/templates pid_file: /run/mailman3/master.pid lock_file: $lock_dir/master.lck [database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://mailman3:<xxxxxxx>@localhost/mailman3?charset=utf8&use_unicode=1 debug: no [logging.debian] format: %(asctime)s (%(process)d) %(message)s datefmt: %b %d %H:%M:%S %Y propagate: no level: info path: mailman.log [webservice] hostname: localhost port: 8001 use_https: no show_tracebacks: yes api_version: 3.1 admin_user: restadmin admin_pass: <xxxxxxx> [mta] incoming: mailman.mta.postfix.LMTP outgoing: mailman.mta.deliver.deliver smtp_host: box2.gtahardware.co.za smtp_port: 465 smtp_user: <xxxxxxx> smtp_pass: <xxxxxxx> smtp_secure_mode: smtps smtp_verify_cert: yes smtp_verify_hostname: yes lmtp_host: lists.fast.za.net lmtp_port: 8024 max_recipients: 10 max_sessions_per_connection: 0 max_delivery_threads: 0 delivery_retry_period: 5d verp_delimiter: + verp_format: ${bounces}+${local}=${domain} verp_confirm_format: $address+$cookie verp_regexp: ^(?P<bounces>[^+]+?)\+(?P<local>[^=]+)=(?P<domain>[^@]+)@.*$ verp_confirm_regexp: ^(.*<)?(?P<addr>[^+]+?)\+(?P<cookie>[^@]+)@.*$ verp_confirmations: yes verp_probe_format: $bounces+$token@$domain verp_probe_regexp: ^(?P<bounces>[^+]+?)\+(?P<token>[^@]+)@.*$ verp_probes: no max_autoresponses_per_day: 10 remove_dkim_headers: no configuration: python:mailman.config.postfix [bounces] register_bounces_every: 15m [archiver.master] class: enable: no configuration: changeme clobber_date: maybe clobber_skew: 1d [archiver.mhonarc] class: mailman.archiving.mhonarc.MHonArc configuration: python:mailman.config.mhonarc [archiver.mail_archive] class: mailman.archiving.mailarchive.MailArchive configuration: python:mailman.config.mail_archive [archiver.prototype] class: mailman.archiving.prototype.Prototype [styles] default: legacy-default [nntp] user: password: host: port: remove_headers: nntp-posting-host nntp-posting-date x-trace x-complaints-to xref date-received posted posting-version relay-version received rewrite_duplicate_headers: To X-Original-To CC X-Original-CC Content-Transfer-Encoding X-Original-Content-Transfer-Encoding MIME-Version X-MIME-Version gatenews_every: 5m [dmarc] resolver_timeout: 3s resolver_lifetime: 5s org_domain_data_url: https://publicsuffix.org/list/public_suffix_list.dat cache_lifetime: 7d [urlpatterns] mailinglist: $base_url/lists/$list_id held_message: $base_url/lists/$list_id/held_messages pending_subscriptions: $base_url/lists/$list_id/subscription_requests pending_unsubscriptions: $base_url/lists/$list_id/unsubscription_requests domain: $base_url/domains/$domain
Why doesn't port 8024 show up then?
On 8/2/24 09:43, Roland Giesler via Mailman-users wrote:
Ah, so I just have to figure out why the LMTP runner is not running.
Stop Mailman core and then start it. You must stop and then start as opposed to restart.
If lmtp runner is not running after that check /var/log/mailman3/mailman.log for possible reasons. You can also check that log for reasons why it isn't currently running.
You can also check /var/log/syslog* for events like out_of_memory that might have killed it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024/08/02 19:02, Mark Sapiro wrote:
On 8/2/24 09:43, Roland Giesler via Mailman-users wrote:
Ah, so I just have to figure out why the LMTP runner is not running.
Stop Mailman core and then start it. You must stop and then start as opposed to restart.
If lmtp runner is not running after that check /var/log/mailman3/mailman.log for possible reasons. You can also check that log for reasons why it isn't currently running.
You can also check /var/log/syslog* for events like out_of_memory that might have killed it.
I do now have the lmtp runner on port 8024.
# netstat -tunap | grep 8024 tcp 0 0 192.168.161.103:8024 0.0.0.0:* LISTEN 46730/python3
In mailman.cfg I set it to listen on the LAN address instead of 172.0.0.1, since I'm expecting mails from my external MTA server.
In the logs of the MTA I see this however: warning: do not list domain fast.za.net in BOTH virtual_mailbox_domains and relay_domains
Mailman creates these entries, but postfix doesn't like it. I don't see any mail delivered to the mailman yet. Is this the problem?
In the MTA postfix main.cf:
relay_domains = hash:/etc/mailman3/data/postfix_domains
cat /etc/mailman3/data/postfix_domains # AUTOMATICALLY GENERATED BY MAILMAN ON 2024-08-05 07:20:17 # # This file is generated by Mailman, and is kept in sync with the binary hash # file. YOU SHOULD NOT MANUALLY EDIT THIS FILE unless you know what you're # doing, and can keep the two files properly in sync. If you screw it up, # you're on your own.
fast.za.net fast.za.net
and also
local_recipient_maps=$virtual_mailbox_maps, hash:/etc/mailman3/data/postfix_lmtp
cat /etc/mailman3/data/postfix_lmtp # AUTOMATICALLY GENERATED BY MAILMAN ON 2024-08-05 07:20:17 # # This file is generated by Mailman, and is kept in sync with the binary hash # file. YOU SHOULD NOT MANUALLY EDIT THIS FILE unless you know what you're # doing, and can keep the two files properly in sync. If you screw it up, # you're on your own.
# Aliases which are visible only in the @fast.za.net domain. test@fast.za.net lmtp:[192.168.161.103]:8024 test-bounces@fast.za.net lmtp:[192.168.161.103]:8024 test-confirm@fast.za.net lmtp:[192.168.161.103]:8024 test-join@fast.za.net lmtp:[192.168.161.103]:8024 test-leave@fast.za.net lmtp:[192.168.161.103]:8024 test-owner@fast.za.net lmtp:[192.168.161.103]:8024 test-request@fast.za.net lmtp:[192.168.161.103]:8024 test-subscribe@fast.za.net lmtp:[192.168.161.103]:8024 test-unsubscribe@fast.za.net lmtp:[192.168.161.103]:8024
Then there's: virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf
cat /etc/postfix/virtual-mailbox-domains.cf dbpath=/home/user-data/mail/users.sqlite query = SELECT 1 FROM users WHERE email LIKE '%%@%s' UNION SELECT 1 FROM aliases WHERE source LIKE '%%@%s' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%%@%s'
When I run that query in sqlite3, it returns no records, so I'm not sure how this is supposed to work. %s to me means that first argument, so is this used in python and then %s is the argument sent to this query?
The bottom line of this all is that a new signup message that I reply to is being received by box2.gtahardware,co.za, but not by mailman3 and I cannot find the point where it fails in the logs.
On 2024/08/05 12:44, Roland Giesler via Mailman-users wrote:
I do now have the lmtp runner on port 8024.
# netstat -tunap | grep 8024 tcp 0 0 192.168.161.103:8024 0.0.0.0:* LISTEN 46730/python3
In mailman.cfg I set it to listen on the LAN address instead of 172.0.0.1, since I'm expecting mails from my external MTA server.
In the logs of the MTA I see this however: warning: do not list domain fast.za.net in BOTH virtual_mailbox_domains and relay_domains
Mailman creates these entries, but postfix doesn't like it. I don't see any mail delivered to the mailman yet. Is this the problem?
I have come to the conclusion that without substantially hacking power-mail-in-a-box to provide the ability to facility forwarding incoming emails to a list (or other) server, mailman3 cannot be used with it as external MTA.
So I have removed my changes and have done "dpkg-reconfigure postfix" on the mailman3 machine to start with a functional postfix server.
Thanks to all that assisted me in this quest. I may revisit this at some future date, since I think Power-MiaB is a excellent service to use for an internet mail server and provides a whole lot of advantages to a mailing list server, but for now, Postfix properly configured manually will have to do.
I see this reply is now moot as you have now configured list mail to go directly to the Mailman server, but ...
On 8/5/24 03:44, Roland Giesler via Mailman-users wrote:
In the logs of the MTA I see this however: warning: do not list domain fast.za.net in BOTH virtual_mailbox_domains and relay_domains
Mailman creates these entries, but postfix doesn't like it. I don't see any mail delivered to the mailman yet. Is this the problem?
Probably not. It is telling you that mail to the fast.za.net domain cannot both be delivered to local mailboxes (virtual_mailbox_domains) and relayed to foreign hosts (relay_domains)
In the MTA postfix main.cf:
relay_domains = hash:/etc/mailman3/data/postfix_domains
cat /etc/mailman3/data/postfix_domains ...
and also
local_recipient_maps=$virtual_mailbox_maps, hash:/etc/mailman3/data/postfix_lmtp
cat /etc/mailman3/data/postfix_lmtp ...
How about
transport_maps = hash:/etc/mailman3/data/postfix_lmtp
Then there's: virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf
cat /etc/postfix/virtual-mailbox-domains.cf dbpath=/home/user-data/mail/users.sqlite query = SELECT 1 FROM users WHERE email LIKE '%%@%s' UNION SELECT 1 FROM aliases WHERE source LIKE '%%@%s' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%%@%s'
When I run that query in sqlite3, it returns no records, so I'm not sure how this is supposed to work. %s to me means that first argument, so is this used in python and then %s is the argument sent to this query?
See https://www.postfix.org/sqlite_table.5.html
%% is replaced with % which is a SQL wildcard matching anything and
%s is replaced by the key postfix is looking for, i.e. the domain that
it is asking about.
So, that query becomes
SELECT 1 FROM users WHERE email LIKE '%@fast.za.net' UNION SELECT 1 FROM aliases WHERE source LIKE '%@fast.za.net' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%@fast.za.net'
I.e, it returns true if any user or alias or auto_alias has an address ending in '@fast.za.net' and if that's true the mail to any '@fast.za.net' address including list mail will be stored locally.
If you really have local users on box2.gtahardware.co.za with addresses '@fast.za.net' and you want to relay list mail to lists '@fast.za.net', you need to see https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht....
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2024/08/05 19:59, Mark Sapiro wrote:
I see this reply is now moot as you have now configured list mail to go directly to the Mailman server, but ...
On 8/5/24 03:44, Roland Giesler via Mailman-users wrote:
In the logs of the MTA I see this however: warning: do not list domain fast.za.net in BOTH virtual_mailbox_domains and relay_domains
Mailman creates these entries, but postfix doesn't like it. I don't see any mail delivered to the mailman yet. Is this the problem?
Probably not. It is telling you that mail to the fast.za.net domain cannot both be delivered to local mailboxes (virtual_mailbox_domains) and relayed to foreign hosts (relay_domains)
Thanks, yes, I have since assumed that to be the case.
In the MTA postfix main.cf:
relay_domains = hash:/etc/mailman3/data/postfix_domains
cat /etc/mailman3/data/postfix_domains ...
and also
local_recipient_maps=$virtual_mailbox_maps, hash:/etc/mailman3/data/postfix_lmtp
cat /etc/mailman3/data/postfix_lmtp ...
How about
transport_maps = hash:/etc/mailman3/data/postfix_lmtp
I can't remove the $virtual_mailbox_maps entry, since Power-mailinbox (PMiaB) uses that. It may make Mailman3 work, but break PMiaB).
Then there's: virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf
cat /etc/postfix/virtual-mailbox-domains.cf dbpath=/home/user-data/mail/users.sqlite query = SELECT 1 FROM users WHERE email LIKE '%%@%s' UNION SELECT 1 FROM aliases WHERE source LIKE '%%@%s' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%%@%s'
When I run that query in sqlite3, it returns no records, so I'm not sure how this is supposed to work. %s to me means that first argument, so is this used in python and then %s is the argument sent to this query?
See https://www.postfix.org/sqlite_table.5.html
%%is replaced with%which is a SQL wildcard matching anything and%sis replaced by the key postfix is looking for, i.e. the domain that it is asking about.So, that query becomes
SELECT 1 FROM users WHERE email LIKE '%@fast.za.net' UNION SELECT 1 FROM aliases WHERE source LIKE '%@fast.za.net' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%@fast.za.net'
I.e, it returns true if any user or alias or auto_alias has an address ending in '@fast.za.net' and if that's true the mail to any '@fast.za.net' address including list mail will be stored locally.
Ah, thank you! I created a ticket at MiaB about this, so I'll post your response there. The %s had be stumped at first, but now it's clear.
If you really have local users on box2.gtahardware.co.za with addresses '@fast.za.net' and you want to relay list mail to lists '@fast.za.net', you need to see https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht....
Thank you for that! From that it seems it may still be possible to use PMiaB as my MTA after, but I'll work through that reference and test it and report back.
On Tue, Aug 6, 2024 at 10:46 AM Roland Giesler via Mailman-users < mailman-users@mailman3.org> wrote:
On 2024/08/05 19:59, Mark Sapiro wrote:
I see this reply is now moot as you have now configured list mail to go directly to the Mailman server, but ...
On 8/5/24 03:44, Roland Giesler via Mailman-users wrote:
In the logs of the MTA I see this however: warning: do not list domain fast.za.net in BOTH virtual_mailbox_domains and relay_domains
Mailman creates these entries, but postfix doesn't like it. I don't see any mail delivered to the mailman yet. Is this the problem?
Probably not. It is telling you that mail to the fast.za.net domain cannot both be delivered to local mailboxes (virtual_mailbox_domains) and relayed to foreign hosts (relay_domains)
Thanks, yes, I have since assumed that to be the case.
In the MTA postfix main.cf:
relay_domains = hash:/etc/mailman3/data/postfix_domains
cat /etc/mailman3/data/postfix_domains ...
and also
local_recipient_maps=$virtual_mailbox_maps, hash:/etc/mailman3/data/postfix_lmtp
cat /etc/mailman3/data/postfix_lmtp ...
How about
transport_maps = hash:/etc/mailman3/data/postfix_lmtp
I can't remove the $virtual_mailbox_maps entry, since Power-mailinbox (PMiaB) uses that. It may make Mailman3 work, but break PMiaB).
Then there's: virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf
cat /etc/postfix/virtual-mailbox-domains.cf dbpath=/home/user-data/mail/users.sqlite query = SELECT 1 FROM users WHERE email LIKE '%%@%s' UNION SELECT 1 FROM aliases WHERE source LIKE '%%@%s' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%%@%s'
When I run that query in sqlite3, it returns no records, so I'm not sure how this is supposed to work. %s to me means that first argument, so is this used in python and then %s is the argument sent to this query?
See https://www.postfix.org/sqlite_table.5.html
%%is replaced with%which is a SQL wildcard matching anything and%sis replaced by the key postfix is looking for, i.e. the domain that it is asking about.So, that query becomes
SELECT 1 FROM users WHERE email LIKE '%@fast.za.net' UNION SELECT 1 FROM aliases WHERE source LIKE '%@fast.za.net' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%@fast.za.net'
I.e, it returns true if any user or alias or auto_alias has an address ending in '@fast.za.net' and if that's true the mail to any '@fast.za.net' address including list mail will be stored locally.
Ah, thank you! I created a ticket at MiaB about this, so I'll post your response there. The %s had be stumped at first, but now it's clear.
If you really have local users on box2.gtahardware.co.za with addresses '@fast.za.net' and you want to relay list mail to lists '@fast.za.net', you need to see
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht... .
Thank you for that! From that it seems it may still be possible to use PMiaB as my MTA after, but I'll work through that reference and test it and report back.
I think that ALL MTAs have the concept of local domains (for which mails are delivered to 'local mailboxes') and remote domains (aka relay domains) for which mail is relayed to another host which has the mailboxes. So in your case box2.gtahardware.co.za (this is FQDN) could be handling local emails, e.g roland@gtahardware.co.za, johndoe@gtahardware.co.za, etc. Those are local, and so gtahardware.co.za is a local domain. However, fast.za.net is a relay domain and all mail to XXX@fast.za.net should be relayed to the MM3 server. If your MTA does not have this concept, then it's either not ready for prime time or it wasn't intended to have such ability.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
On 2024/08/06 10:46, Odhiambo Washington via Mailman-users wrote:
On Tue, Aug 6, 2024 at 10:46 AM Roland Giesler via Mailman-users < mailman-users@mailman3.org> wrote:
On 2024/08/05 19:59, Mark Sapiro wrote:
If you really have local users on box2.gtahardware.co.za with addresses '@fast.za.net' and you want to relay list mail to lists '@fast.za.net', you need to see https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht... .
Thank you for that! From that it seems it may still be possible to use PMiaB as my MTA after, but I'll work through that reference and test it and report back. I think that ALL MTAs have the concept of local domains (for which mails are delivered to 'local mailboxes') and remote domains (aka relay domains) for which mail is relayed to another host which has the mailboxes. So in your case box2.gtahardware.co.za (this is FQDN) could be handling local emails, e.groland@gtahardware.co.za,johndoe@gtahardware.co.za, etc. Those are local, and so gtahardware.co.za is a local domain. However, fast.za.net is a relay domain and all mail toXXX@fast.za.net should be relayed to the MM3 server. If your MTA does not have this concept, then it's either not ready for prime time or it wasn't intended to have such ability.
With Mail-in-a-box and it's fork Power-mail-in-a-box, the recipient domain is created when an email address is added for which the box should receive mail. When the last email address for a domain is deleted, the domain is deleted too. More than that, the DNS zone file for the domain is also created and so are certificates, aliases and more. If I stick to that, then it's easy to run a mail server and everything *just works*. This is why I don't want to remove any of the MiaB settings.
If I therefore want to allow mail for fast.za.net onto box2.gtahardware.co.za, then I need to have at least one mailbox for that domain, or an email alias on that box. This of course complicates matters for forwarding, since not all mail for fast.za.net must be forwarded, only the mail that is not specifically configured on box2.
With Mark's link I think I may get this to work, but I need to still impliment and test this. It will take a couple of days for me to get around to it bit by bit (no pun intended!).
On Fri, Aug 2, 2024 at 5:51 PM Roland Giesler via Mailman-users < mailman-users@mailman3.org> wrote:
On 2024/08/01 10:09, Roland Giesler via Mailman-users wrote:
I needed to put these files at these locations:
transport_maps = hash:/etc/mailman3/data/postfix_lmtp local_recipient_maps = hash:/etc/mailman3/data/postfix_lmtp relay_domains = hash:/etc/mailman3/data/postfix_domains
However, since I'm using power-mailinabox, Postfix is configured to use sqlite already and has these lines by default:
virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf virtual_mailbox_maps=sqlite:/etc/postfix/virtual-mailbox-maps.cf virtual_alias_maps=sqlite:/etc/postfix/virtual-alias-maps.cf local_recipient_maps=$virtual_mailbox_maps
So I have only added this line to postfix's main.cf:
# Add postfix_domains path so mailman can add domains to it for lists relay_domains=hash://etc/mailman3/data/postfix_domains
The local_recipient_maps are set by power-mailinabox already, so I'm ignoring what mailman creates.
It seems that this is a mistake. If I don't let postfix read the local_recipient_maps that mailman3 created, the incoming emails are not sent to the other host. So I have a conflict with the local_recipient_maps that postfix has (created by Power-mailinabox) and those created by mailman3.
I'm hoping that if I specify local_recipient_maps more than once and then both sources will be considered. Would that work?
If not, then I'm pretty much stumped, unless I go for the database option that has not been merged yet. Even if I do that, I would need to make sure that mailman3 doesn't remove the other entries in the database which, judging by what I've seen, seems to be the case.
Any ideas?
Quoting the official documentation: https://docs.mailman3.org/en/latest/install/virtualenv.html#setup-mta
If your current main.cf contains settings for these items, including the
default setting for local_recipient_maps, you should add the Mailman
setting to the existing setting rather than replacing it.
Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
participants (3)
-
Mark Sapiro -
Odhiambo Washington -
Roland Giesler