Mixing Mailman 3 with AWS SES
I am moving the website www.t10.org from a private/corporate VM to AWS EC2s.
For many years, t10.org has had very happy results with using Mailman 2 for its email reflector, and I have no real concerns about moving to Mailman 3. My bugbear is the fact that I need the following mixed bag of incoming email addresses to work in concert with each other to receive and handle emails.
t10@t10.org (Primary Mailman Reflector) chair@t10.org (Mailman Reflector to direct messages to the T10 chair and his designates)
- docs@t10.org (document posting mechanics - serviced by a small mountain of home-grown code code)
- bbs@t10.org (other publicly accessible services - serviced by code that recalls the days when everything as done on a Bulletin Board System, BBS)
My best guess is that the AWS Simple Email Service (SES) needs to sit in front of both Mailman and the home-grown code, to properly direct the incoming emails to the right places. The big worry is as follows...
All available evidence read to date suggests that the default AWS installation of Mailman 3 assumes that Mailman 3 is the *only* receiver of emails in the configuration. If true, then putting SES in front of Mailman could be a Herculean challenge.
Or... Hercules may have already cleaned the Augean Stables, and left behind a AWS Lambda function that can serve as a keystone puzzle piece for solving my dilemma.
Thanks for taking the time to look at this.
All the best, .Ralph
FWIW Information about the current t10.org Mailman reflector can be found at https://www.t10.org/t10r.htm. T10 is primarily an open-to-the-public activity, so there are no company confidential secrets there.
For those interested in the archives associated with the t10.org Mailman reflector, they are available at https://www.t10.org/pipermail/t10/ and "inquiring minds" may puzzle over the fact that the archives date back to December 1993. When Mailman 2 was installed, we converted all the Majordomo archives to Mailman 2 format.
Hi Ralph,
On Mar 17, 2021, at 5:14 PM, Ralph Weber <roweber@ieee.org> wrote:
I am moving the website www.t10.org from a private/corporate VM to AWS EC2s.
For many years, t10.org has had very happy results with using Mailman 2 for its email reflector, and I have no real concerns about moving to Mailman 3. My bugbear is the fact that I need the following mixed bag of incoming email addresses to work in concert with each other to receive and handle emails.
t10@t10.org (Primary Mailman Reflector) chair@t10.org (Mailman Reflector to direct messages to the T10 chair and his designates)
- docs@t10.org (document posting mechanics - serviced by a small mountain of home-grown code code)
- bbs@t10.org (other publicly accessible services - serviced by code that recalls the days when everything as done on a Bulletin Board System, BBS)
I am not entirely sure what a “Reflector” really is.
My best guess is that the AWS Simple Email Service (SES) needs to sit in front of both Mailman and the home-grown code, to properly direct the incoming emails to the right places. The big worry is as follows...
All available evidence read to date suggests that the default AWS installation of Mailman 3 assumes that Mailman 3 is the *only* receiver of emails in the configuration. If true, then putting SES in front of Mailman could be a Herculean challenge.
Which specific AWS installation guide are you referring to?
Mailman doesn’t really care about being the only receiver of emails. Mailman can receive emails via LMTP or HTTP protocols. If you can do either of those two, you should be good. I don’t remember offhand someone sharing SES configs/setups for receiving emails, but maybe if someone did get that working, hopefully they’ll chime in.
Sending out emails with SES is easy, you just need to setup the right SMTP server in Mailman’s config.
-- thanks, Abhilash Raj (maxking)
Ralph Weber <roweber@ieee.org> wrote:
I am moving the website www.t10.org from a private/corporate VM to AWS EC2s.
For many years, t10.org has had very happy results with using Mailman 2 for its email reflector, and I have no real concerns about moving to Mailman 3. My bugbear is the fact that I need the following mixed bag of incoming email addresses to work in concert with each other to receive and handle emails.
[...]
My best guess is that the AWS Simple Email Service (SES) needs to sit in front of both Mailman and the home-grown code, to properly direct the incoming emails to the right places. The big worry is as follows...
Why do you believe that is the case? In your current setup your mail server acts as primary MX, the server runs an MTA with an interface to Mailman. Why would you believe this to be different in the AWS environment?
All available evidence read to date suggests that the default AWS installation of Mailman 3 assumes that Mailman 3 is the *only* receiver of emails in the configuration. If true, then putting SES in front of Mailman could be a Herculean challenge.
Which instructions are these?
When I read about using Mailman on AWS, my experience is that getting email into the system is fine because AWS allows incoming connections on port 25, and so long as you have a static public IP address attached to the instance that routes traffic correctly incoming mail is fine. The challenge is with outbound mail from your server, as directly sending outbound email from AWS IP addresses is an issue because the AWS Ips are listed on many spam lists as spammers have been known to set up instances and abuse them by sending out bulk mail. Sending outbound emails via the SES service from Mailman is possible, but it comes with some caveats due to the way that SES validates email addresses on the platform. Some users have been able to get it working on this list, but have had to put the lists into an anonymous mode where the email of the sender is obfuscated so as not to trip up SES.
For this reason I haven't moved my Mailman instances onto the public cloud, I instead use a VPS provider who's Ips have a good sending reputation. I am still interested in doing this work however, just need to work out the best option for sending out the email effectively.
Thanks. Andrew.
Ralph Weber writes:
I am moving the website www.t10.org from a private/corporate VM to AWS EC2s.
For many years, t10.org has had very happy results with using Mailman 2 for its email reflector, and I have no real concerns about moving to Mailman 3. My bugbear is the fact that I need the following mixed bag of incoming email addresses to work in concert with each other to receive and handle emails.
t10@t10.org (Primary Mailman Reflector) chair@t10.org (Mailman Reflector to direct messages to the T10 chair and his designates)
- docs@t10.org (document posting mechanics - serviced by a small mountain of home-grown code code
- bbs@t10.org (other publicly accessible services - serviced by code that recalls the days when everything as done on a Bulletin Board System, BBS)
This isn't a Mailman question, to be frank. As Abhilash points out, Mailman doesn't care about other mailboxes served by its host (unless they conflict with list names or adinistrative addresses derived from list names, which evidently yours do not). It's a question about the mail service at AWS. I've not seen any documentation in our project about setting up at AWS; I don't know anybody who has done so. (I haven't looked for either though.)
I took a quick look for AWS SES documentation. They don't seem to have any real documentation online for the general public, only lists of features. They also seem to be very focused on sending rather than receiving. Much talk of how much mail you can send, bulk emailing, statistics on responses, and the like. None about setting up for incoming mail.
Probably I could get access to online help if I signed up for a free account, but that's above my pay grade.
My best guess is that the AWS Simple Email Service (SES) needs to sit in front of both Mailman and the home-grown code,
I don't know if AWS provides an alternative Complicated Email Service, but yes, something like SES is going to have to sit in front of the various mailboxes.
Mailman itself has a certain amount of capability to deal with the main email protocols, but it is definitely not capable of replacing the main mail server. For security reasons I would definitely advise against any Mailman 3 presence directly on the Internet. It was designed on the assumption that it would talk to the Internet via a standard mail server, Postorius (for administration), and HyperKitty (for mail archives)[1], and occasionally to admins logging in with shell accounts.
All available evidence read to date
URLs, please. As I wrote above, Amazon's documentation is at best not very discoverable.
suggests that the default AWS installation
Whose default installation of Mailman? AWS's? Typical 3rd party configurations?
of Mailman 3 assumes that Mailman 3 is the *only* receiver of emails in the configuration.
That's possible, but it's likely an artifact of the application -- I would expect that many who host Mailman on AWS do so to isolate Mailman from other services they provide. So it seems reasonable that in many such installations Mailman is *the* service for that instance.
But mail servers are designed to be many inputs, many outputs services. If SES itself is as restricted as you fear, surely AWS provides an alternative, more powerful mail service. I can imagine quite a few restrictions, other than the backing application, on the SES that Amazon *might* impose. But without access to documentation, we can't say much.
Or... Hercules may have already cleaned the Augean Stables, and left behind a AWS Lambda function that can serve as a keystone puzzle piece for solving my dilemma.
Good luck on that. I don't know of any, but that would be a quick solution for you. I hope somebody here knows more. As evidenced by Andrew's post, there are folks thinking about it, so maybe somebody's actually tried -- anybody? Successes? Pitfalls to watch for?
The other issue Andrew mentions could be important. Since Solar Winds I'm seeing daily attempts to access my O365 server (nonexistent) from AWS. Given the size of AWS, I think they probably actually do a good job of keeping felonious behavior to a minimum[2], but a lot of folks I know are happy to cut off huge netblocks until somebody they know actually complains. :-(
Footnotes: [1] Or more precisely, some webservice providing those services mediating between the Internet and Mailman sitting cozily behind a firewall.
[2] DigitalOcean, on the other hand, is accumulating blocked /24s, and even /16s, at my firewall. But that's another story.
To close the loop on this topic...
Comments received directly from one of the people who developed the AWS Marketplace kits for Mailman 3 and Mailman 4 have definitively established that AWS Mailman is *not* intended to interact with AWS SES when receiving email messages.
AWS Mailman is designed to receive its email messages via Postfix.
With many thanks to all those on this list who corroborated "Mailman and SES don't mix" mantra here, we are taking the advice received from so many sources and ending efforts to do the impossible.
All the best, .Ralph
On 3/20/2021 8:23 AM, Stephen J. Turnbull wrote:
Ralph Weber writes:
I am moving the website www.t10.org from a private/corporate VM to AWS EC2s.
For many years, t10.org has had very happy results with using Mailman 2 for its email reflector, and I have no real concerns about moving to Mailman 3. My bugbear is the fact that I need the following mixed bag of incoming email addresses to work in concert with each other to receive and handle emails.
t10@t10.org (Primary Mailman Reflector) chair@t10.org (Mailman Reflector to direct messages to the T10 chair and his designates)
- docs@t10.org (document posting mechanics - serviced by a small mountain of home-grown code code
- bbs@t10.org (other publicly accessible services - serviced by code that recalls the days when everything as done on a Bulletin Board System, BBS)
This isn't a Mailman question, to be frank. As Abhilash points out, Mailman doesn't care about other mailboxes served by its host (unless they conflict with list names or adinistrative addresses derived from list names, which evidently yours do not). It's a question about the mail service at AWS. I've not seen any documentation in our project about setting up at AWS; I don't know anybody who has done so. (I haven't looked for either though.)
I took a quick look for AWS SES documentation. They don't seem to have any real documentation online for the general public, only lists of features. They also seem to be very focused on sending rather than receiving. Much talk of how much mail you can send, bulk emailing, statistics on responses, and the like. None about setting up for incoming mail.
Probably I could get access to online help if I signed up for a free account, but that's above my pay grade.
My best guess is that the AWS Simple Email Service (SES) needs to sit in front of both Mailman and the home-grown code,
I don't know if AWS provides an alternative Complicated Email Service, but yes, something like SES is going to have to sit in front of the various mailboxes.
Mailman itself has a certain amount of capability to deal with the main email protocols, but it is definitely not capable of replacing the main mail server. For security reasons I would definitely advise against any Mailman 3 presence directly on the Internet. It was designed on the assumption that it would talk to the Internet via a standard mail server, Postorius (for administration), and HyperKitty (for mail archives)[1], and occasionally to admins logging in with shell accounts.
All available evidence read to date
URLs, please. As I wrote above, Amazon's documentation is at best not very discoverable.
suggests that the default AWS installation
Whose default installation of Mailman? AWS's? Typical 3rd party configurations?
of Mailman 3 assumes that Mailman 3 is the *only* receiver of emails in the configuration.
That's possible, but it's likely an artifact of the application -- I would expect that many who host Mailman on AWS do so to isolate Mailman from other services they provide. So it seems reasonable that in many such installations Mailman is *the* service for that instance.
But mail servers are designed to be many inputs, many outputs services. If SES itself is as restricted as you fear, surely AWS provides an alternative, more powerful mail service. I can imagine quite a few restrictions, other than the backing application, on the SES that Amazon *might* impose. But without access to documentation, we can't say much.
Or... Hercules may have already cleaned the Augean Stables, and left behind a AWS Lambda function that can serve as a keystone puzzle piece for solving my dilemma.
Good luck on that. I don't know of any, but that would be a quick solution for you. I hope somebody here knows more. As evidenced by Andrew's post, there are folks thinking about it, so maybe somebody's actually tried -- anybody? Successes? Pitfalls to watch for?
The other issue Andrew mentions could be important. Since Solar Winds I'm seeing daily attempts to access my O365 server (nonexistent) from AWS. Given the size of AWS, I think they probably actually do a good job of keeping felonious behavior to a minimum[2], but a lot of folks I know are happy to cut off huge netblocks until somebody they know actually complains. :-(
Footnotes: [1] Or more precisely, some webservice providing those services mediating between the Internet and Mailman sitting cozily behind a firewall.
[2] DigitalOcean, on the other hand, is accumulating blocked /24s, and even /16s, at my firewall. But that's another story.
On 3/24/21 9:54 AM, Ralph Weber wrote:
Comments received directly from one of the people who developed the AWS Marketplace kits for Mailman 3 and Mailman 4 have definitively established that AWS Mailman is *not* intended to interact with AWS SES when receiving email messages
I don't think there is a Mailman 4 and the person who is marketing his services under that name is sending a confusing message to the public.
-- Brian Carpenter Harmonylists.com Emwd.com
On 3/24/21 7:58 AM, Brian Carpenter wrote:
On 3/24/21 9:54 AM, Ralph Weber wrote:
Comments received directly from one of the people who developed the AWS Marketplace kits for Mailman 3 and Mailman 4 have definitively established that AWS Mailman is *not* intended to interact with AWS SES when receiving email messages
I don't think there is a Mailman 4 and the person who is marketing his services under that name is sending a confusing message to the public.
Danil Smirnov owns both the mailman3.com and mailman4.com domains. Mailman 4 is apparently his branding of "Mailman 3 in AWS Marketplace". In response to our request, the following is included at the bottom of <https://mailman3.com/>:
Mailman3.com (this site) is unaffiliated with Mailman3.org, the domain used to coordinate development of the Mailman 3 free software suite. The Mailman 3 core developers are not involved with Mailman3.com, and cannot answer any questions or provide any support related to its operations. If you are looking for resources for development of the Mailman 3 free software suite, please see mailman3.org.
Yes, this is all ripe for confusion ...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Wed, 2021-03-24 at 10:57 -0700, Mark Sapiro wrote:
On 3/24/21 7:58 AM, Brian Carpenter wrote:
On 3/24/21 9:54 AM, Ralph Weber wrote:
Comments received directly from one of the people who developed the AWS Marketplace kits for Mailman 3 and Mailman 4 have definitively established that AWS Mailman is *not* intended to interact with AWS SES when receiving email messages
I don't think there is a Mailman 4 and the person who is marketing his services under that name is sending a confusing message to the public.
Danil Smirnov owns both the mailman3.com and mailman4.com domains. Mailman 4 is apparently his branding of "Mailman 3 in AWS Marketplace". In response to our request, the following is included at the bottom of <https://mailman3.com/>;:
Mailman3.com (this site) is unaffiliated with Mailman3.org, the domain used to coordinate development of the Mailman 3 free software suite. The Mailman 3 core developers are not involved with Mailman3.com, and cannot answer any questions or provide any support related to its operations. If you are looking for resources for development of the Mailman 3 free software suite, please see mailman3.org.
Yes, this is all ripe for confusion ...
Just to add, Danil also owns mailman2.com, and I own (and parked) mailman2.org.
-Jim P.
On 3/24/21 2:00 PM, Jim Popovitch via Mailman-users wrote:
Just to add, Danil also owns mailman2.com, and I own (and parked) mailman2.org.
-Jim P.
That's it! I am buying Mailman5, Mailman6, and Mailman7 in .com/.org/.mailman. I am also purchasing Mailman2.1.com.
-- Brian Carpenter Harmonylists.com Emwd.com
On 3/24/21 2:00 PM, Jim Popovitch via Mailman-users wrote:
Danil Smirnov owns both the mailman3.com and mailman4.com domains. Mailman 4 is apparently his branding of "Mailman 3 in AWS Marketplace". In response to our request, the following is included at the bottom of <https://mailman3.com/>;:
Has anyone asked Danil if he is willing to hand over the mailman3.com domain to the Mailman 3 development team? I think that would be a nice gesture. Same with Mailman4.com.
-- Brian Carpenter Harmonylists.com Emwd.com
On Wed, 2021-03-24 at 14:00 -0400, Jim Popovitch via Mailman-users wrote:
On Wed, 2021-03-24 at 10:57 -0700, Mark Sapiro wrote:
On 3/24/21 7:58 AM, Brian Carpenter wrote:
On 3/24/21 9:54 AM, Ralph Weber wrote:
Comments received directly from one of the people who developed the AWS Marketplace kits for Mailman 3 and Mailman 4 have definitively established that AWS Mailman is *not* intended to interact with AWS SES when receiving email messages
I don't think there is a Mailman 4 and the person who is marketing his services under that name is sending a confusing message to the public.
Danil Smirnov owns both the mailman3.com and mailman4.com domains. Mailman 4 is apparently his branding of "Mailman 3 in AWS Marketplace". In response to our request, the following is included at the bottom of <https://mailman3.com/>;;:
Mailman3.com (this site) is unaffiliated with Mailman3.org, the domain used to coordinate development of the Mailman 3 free software suite. The Mailman 3 core developers are not involved with Mailman3.com, and cannot answer any questions or provide any support related to its operations. If you are looking for resources for development of the Mailman 3 free software suite, please see mailman3.org.
Yes, this is all ripe for confusion ...
Just to add, Danil also owns mailman2.com, and I own (and parked) mailman2.org.
Apologies to this list and Danil who has informed me that mailman2.com is not theirs. This of course means that there is another stakeholder out there somewhere.... ;)
-Jim P.
On 3/25/21 2:51 PM, Jim Popovitch via Mailman-users wrote:
Apologies to this list and Danil who has informed me that mailman2.com is not theirs. This of course means that there is another stakeholder out there somewhere....;)
Would be nice if Danil could let the list know and in that same communication, he can explain why he is promoting Mailman 4 when there is no Mailman 4. We can then move to the discussion of him charitably giving mailman3.com to the Mailman 3 developers. How about it Danil?
-- Brian Carpenter Harmonylists.com Emwd.com
Hey Brian!
On Thu, Mar 25, 2021 at 10:36 PM Brian Carpenter <brian_carpenter@emwd.com> wrote:
On 3/25/21 2:51 PM, Jim Popovitch via Mailman-users wrote:
Apologies to this list and Danil who has informed me that mailman2.com is not theirs. This of course means that there is another stakeholder out there somewhere....;)
Would be nice if Danil could let the list know and in that same communication, he can explain why he is promoting Mailman 4 when there is no Mailman 4. We can then move to the discussion of him charitably giving mailman3.com to the Mailman 3 developers. How about it Danil?
This is a great question. Thank you.
No, there are no plans to pass domains I own to you, or any other party (unless you offer a very good price, of course). They are in use of commercial services, which are based on the Mailman 3 open-source software and listed in the directory of Mailman hosting services <https://wiki.list.org/COM/Mailman%20hosting%20services> (just as your company).
*Mailman3.com* is Mailman 3 commercial hosting service (that, I think, is perfectly reflected in the domain name). It uses a shared hosting approach, which allows you to get a full-fledged Mailman 3 server for as little as *8.25 EUR* (monthly fee).
Mailman3.com has also declared an open-source support initiative to provide a *free Mailman 3 hosting to OSS projects*. (To apply, send the OSS project URL and approximate monthly volume to hostmaster@mailman3.com.)
*Mailman4.com* is a marketing initiative to emphasize the unique possibility to install *Mailman 3 in AWS* infrastructure with a few clicks to leverage all the features of AWS Marketplace, AWS customer protection, and pay-as-you-go billing scheme.
(Being a bit provocative, the initiative was named while taking into consideration that the fourth version of Mailman does not exist yet and is hardly expected soon. After the version released, the appropriate business decision will be undertaken immediately.)
With Mailman4.com you get your Mailman 3 server in the dedicated EC2 instance for the same price which you would pay if you deploy it to your own EC2. *Reliable mail relay*, which is carefully groomed to fit into email delivery best practices, is *included for free*.
Mailman4.com is especially useful for *educational and evaluation purposes* as it allows to deploy a fully functional Mailman 3 server just in one minute and be charged *from 2 to 4 cents hourly* (depend on Hyperkitty presence). And then cancel it instantly when not needed anymore.
For both Mailman3.com and Mailman4.com services it's possible to develop a *custom integration of Mailman Core API with third-party services* via publicly (but securely) exposed endpoints - the option which some of our customers are enjoying already.
For example, one might create mailing lists and manage their subscribers list programmatically in order to sync it with some internal user directory. (Performance issues, which occured when a number of users is large, have been discovered and fixed already.)
The existence of the services, described above, is already bringing benefits to Mailman 3 development community in several areas, i.e. additional paid promotion of the software (e.g to attract more GSoC students), extensive testing in the real world (which is particularly useful for dockerized setup), and contributions to the code.
I hope this helps and best luck with *your* Mailman business.
With my best regards, Danil Smirnov Mailman3.com proud owner
-- Brian Carpenter Harmonylists.com Emwd.com
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Danil Smirnov wrote:
[...]
*Mailman4.com* is a marketing initiative to emphasize the unique possibility to install *Mailman 3 in AWS* infrastructure with a few clicks to leverage all the features of AWS Marketplace, AWS customer protection, and pay-as-you-go billing scheme.
Getting this back on topic, would you be willing to share how you are sending outbound mail through AWS servers without it being marked as spam by nearly every blacklist out there? I know AWS let you set reverse DNS on elastic Ips, however whenever I send outbound mail via MX lookups directly on AWS servers it always gets blacklisted. My solution has been to use an expensive outbound relay.
Any suggestions on how you are doing this for users who purchase your Mailman marketplace image?
Thanks. Andrew.
Hi Andrew,
пт, 26 мар. 2021 г., 23:29 Andrew Hodgson <andrew@hodgson.io>:
Danil Smirnov wrote:
[...]
*Mailman4.com* is a marketing initiative to emphasize the unique possibility to install *Mailman 3 in AWS* infrastructure with a few clicks to leverage all the features of AWS Marketplace, AWS customer protection, and pay-as-you-go billing scheme.
Getting this back on topic, would you be willing to share how you are sending outbound mail through AWS servers without it being marked as spam by nearly every blacklist out there? I know AWS let you set reverse DNS on elastic Ips, however whenever I send outbound mail via MX lookups directly on AWS servers it always gets blacklisted. My solution has been to use an expensive outbound relay.
Any suggestions on how you are doing this for users who purchase your Mailman marketplace image?
I suspect you are talking about ESPs marking incoming mails as spam rather than anti-spam blacklists here.
I've posted an article on the topic a while ago: https://blog.smirnov.la/how-to-send-emails-in-2020-3ad76a31a7cc
Also, there is a "warm-up" of a new IP for some ESPs (but not for all of them) - 1-2 weeks - that you just need to wait out.
Best, Danil
Thanks. Andrew.
Danil Smirnov wrote:
пт, 26 мар. 2021 г., 23:29 Andrew Hodgson <andrew@hodgson.io>:
Danil Smirnov wrote:
[...]
Getting this back on topic, would you be willing to share how you are sending outbound mail through AWS servers without it being marked as spam by nearly every blacklist out there?
[...]
I suspect you are talking about ESPs marking incoming mails as spam rather than anti-spam blacklists here.
I've posted an article on the topic a while ago:
Thanks for this, I may try and get my VPS onto AWS in the coming months. I note Azure now let you set reverse DNS as well.
Andrew.
participants (8)
-
Abhilash Raj -
Andrew Hodgson -
Brian Carpenter -
Danil Smirnov -
Jim Popovitch -
Mark Sapiro -
Ralph Weber -
Stephen J. Turnbull