How to Increase outgoing mail volume for Mailman 3
Hello everyone,
We had a new client come on board that brought a list that had the largest out-going mail volume that I have ever dealt with. So I did what any sane server administrator would do that is running Mailman 3: I ran to Mark Sapiro for help! Here is what the Sage of all things Mailman counseled me to do:
- Disabled VERP delivery. VERP can help improve mail delivery by delivering 1 message per SMTP transaction (I think I got that right). Using VERP is counter-productive as it can seriously delay outgoing mail delivery for a high-volume list. So you can use something like the following in the MTA section of your mailman.cfg:
[mta] verp_confirmations: no verp_personalized_deliveries: no verp_delivery_interval: 0
- I configured Mailman 3 to run 4 outgoing runners. The default is 1. This action was the game changer. It allows Mailman to handle 4 incoming messages at a time instead of 1. My client's incoming list volume was high enough where it was overwhelming the single outgoing runner. Here is what I added to the mailman.cfg file:
[runner.out] instances: 4
Doing this step immediately solved the backlog in var/queue/out.
- If you start up with a new server for a list that is high volume then keep an eye out for Proofpoint and Vade Security as they will block your new IP address quicker than Mark Sapiro can say "This is a bug. See <some Gitlab.com url>"
Mark, thank you!
-- Brian Carpenter Harmonylists.com Emwd.com
On Nov 10, 2020, at 12:30 PM, Brian Carpenter <brian_carpenter@emwd.com> wrote:
- Disabled VERP delivery. VERP can help improve mail delivery by delivering 1 message per SMTP transaction (I think I got that right). Using VERP is counter-productive as it can seriously delay outgoing mail delivery for a high-volume list. So you can use something like the following in the MTA section of your mailman.cfg:
[mta] verp_confirmations: no verp_personalized_deliveries: no verp_delivery_interval: 0
The downside to this is that you lose automated bounce processing, of course.
- Mark
mark@pdc-racing.net | 408-348-2878
On 11/10/20 1:02 PM, Mark Dadgar wrote:
The downside to this is that you lose automated bounce processing, of course.
You don't lose bounce processing. It just becomes a bit less reliable. The VERPed envelope sender says exactly what address the message was sent to. Without this information, Mailman (actually flufl.bounce <https://fluflbounce.readthedocs.io/en/latest/README.html>) has to parse the returned bounce message to determine the bouncing address. There are many MTAs that do not send RFC 1894 compliant DSNs and we have several heuristic recognizers to try to parse these, but they can fail.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Nov 10, 2020, at 1:18 PM, Mark Sapiro <mark@msapiro.net> wrote:
The downside to this is that you lose automated bounce processing, of course.
You don't lose bounce processing. It just becomes a bit less reliable. The VERPed envelope sender says exactly what address the message was sent to. Without this information, Mailman (actually flufl.bounce <https://fluflbounce.readthedocs.io/en/latest/README.html>) has to parse the returned bounce message to determine the bouncing address. There are many MTAs that do not send RFC 1894 compliant DSNs and we have several heuristic recognizers to try to parse these, but they can fail.
Brian specified:
verp_personalized_deliveries: no
Doesn’t that default back to regular mass delivery, as opposed to individualized VERPed envelopes, which mandates individual deliveries?
- Mark
mark@pdc-racing.net | 408-348-2878
On 11/10/20 1:22 PM, Mark Dadgar wrote:
Brian specified:
verp_personalized_deliveries: no
Doesn’t that default back to regular mass delivery, as opposed to individualized VERPed envelopes, which mandates individual deliveries?
That depends on the setting of verp_delivery_interval. If verp_delivery_interval = 1, every message will be VERPed regardless of verp_personalized_deliveries. If verp_delivery_interval = 0 and verp_personalized_deliveries = no, deliveries won't be VERPed as you note. If verp_delivery_interval = 0 and verp_personalized_deliveries = yes, only personalized deliveries (i.e. individual messages, not digests, from lists with Personalization enabled) will be VERPed.
The theory is VERPing has a cost because it requires a separate message sent to each recipient rather than a single message with multiple recipients, but personalization also requires a separate message sent to each recipient, so those can be VERPed at no extra cost.
But to be clear, automated bounce processing does not require VERP. It just works a tiny bit more reliably with VERP.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 11/10/20 4:40 PM, Mark Sapiro wrote:
On 11/10/20 1:22 PM, Mark Dadgar wrote:
Brian specified:
verp_personalized_deliveries: no
Doesn’t that default back to regular mass delivery, as opposed to individualized VERPed envelopes, which mandates individual deliveries?
That depends on the setting of verp_delivery_interval. If verp_delivery_interval = 1, every message will be VERPed regardless of verp_personalized_deliveries. If verp_delivery_interval = 0 and verp_personalized_deliveries = no, deliveries won't be VERPed as you note. If verp_delivery_interval = 0 and verp_personalized_deliveries = yes, only personalized deliveries (i.e. individual messages, not digests, from lists with Personalization enabled) will be VERPed.
The theory is VERPing has a cost because it requires a separate message sent to each recipient rather than a single message with multiple recipients, but personalization also requires a separate message sent to each recipient, so those can be VERPed at no extra cost.
But to be clear, automated bounce processing does not require VERP. It just works a tiny bit more reliably with VERP.
We actually enable VERP by default on all new installations of Mailman 3 as we don't see a real performance hit and its benefits are generally very good. This new list however was a whole different beast. Also due to the nature of the list, the benefits of VERPing pales in comparison to the timeliness of the mail being sent.
--
Brian Carpenter Harmonylists.com Emwd.com
On Nov 10, 2020, at 1:40 PM, Mark Sapiro <mark@msapiro.net> wrote:
Brian specified:
verp_personalized_deliveries: no
Doesn’t that default back to regular mass delivery, as opposed to individualized VERPed envelopes, which mandates individual deliveries?
That depends on the setting of verp_delivery_interval. If verp_delivery_interval = 1, every message will be VERPed regardless of verp_personalized_deliveries. If verp_delivery_interval = 0 and verp_personalized_deliveries = no, deliveries won't be VERPed as you note. If verp_delivery_interval = 0 and verp_personalized_deliveries = yes, only personalized deliveries (i.e. individual messages, not digests, from lists with Personalization enabled) will be VERPed.
The theory is VERPing has a cost because it requires a separate message sent to each recipient rather than a single message with multiple recipients, but personalization also requires a separate message sent to each recipient, so those can be VERPed at no extra cost.
But to be clear, automated bounce processing does not require VERP. It just works a tiny bit more reliably with VERP.
Great explanation - thank you.
- Mark
mark@pdc-racing.net | 408-348-2878
On 11/10/20 12:30 PM, Brian Carpenter wrote:
Hello everyone,
We had a new client come on board that brought a list that had the largest out-going mail volume that I have ever dealt with. So I did what any sane server administrator would do that is running Mailman 3: I ran to Mark Sapiro for help! Here is what the Sage of all things Mailman counseled me to do:
Thank you <blush>
- Disabled VERP delivery. VERP can help improve mail delivery by delivering 1 message per SMTP transaction (I think I got that right). Using VERP is counter-productive as it can seriously delay outgoing mail delivery for a high-volume list. So you can use something like the following in the MTA section of your mailman.cfg:
Actually, I mentioned VERP, but I don't think I recommended disabling it. Also note that these settings:
[mta] verp_confirmations: no verp_personalized_deliveries: no verp_delivery_interval: 0 are the defaults. Further, verp_confirmations is a holdover from MM 2.1 and has no effect, although prior to Mailman 3.3.2 confirmations were sent with
confirm <token>as the Subject: and beginning with Mailman 3.3.2, confirmations are sent withYour confirmation is needed ...as the Subject:.
Actually, I think enabling VERP with
[mta] verp_delivery_interval: 1
is generally a good idea for more reliable bounce detection.
- I configured Mailman 3 to run 4 outgoing runners. The default is 1. This action was the game changer. It allows Mailman to handle 4 incoming messages at a time instead of 1. My client's incoming list volume was high enough where it was overwhelming the single outgoing runner. Here is what I added to the mailman.cfg file:
[runner.out] instances: 4
Doing this step immediately solved the backlog in var/queue/out.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Brian Carpenter -
Mark Dadgar -
Mark Sapiro