Avoiding non-members to be saved.
Hi all:
Yesterday I had a problem with a private list. A spammer sent a mail forging one address that was a non-member (in fact was an "accept this no subscriber mails" from mailman 2.1 import) with very bad intentions and it was distributed. There were no list of "accept this non members" in the Postorius config (but I'm sure that that address was "accepted" in 2.1). To avoid this from any other non member address I deleted ALL non-members.
I think it is solved now (deleting all and putting to hold just the non-members addresses that I would want to pass messages on)
The fact is that any sender address of mails sent to the list is saved as a "non member", and it make me nervous as it will grow despite it is a legitimate message or not.
Anybody knows if there is a per list config to avoid saving all that addresses as non members? o a overall config?
I couldn't find anything about this in the config pages (or googling it)
Thanks in advance.
--
Guillermo Hernandez (Oldno7) via Mailman-users writes:
Yesterday I had a problem with a private list. A spammer sent a mail forging one address that was a non-member (in fact was an "accept this no subscriber mails" from mailman 2.1 import) with very bad intentions and it was distributed.
If you upgrade from Mailman 2 to Mailman 3, the accept_these_nonmembers list for each list is grandfathered into the Mailman 3 list (from rules/moderation.py):
# Check the '*_these_nonmembers' properties first. XXX These are
# legacy attributes from MM2.1; their database type is 'pickle' and
# they should eventually get replaced.
for action_name in ('accept', 'hold', 'reject', 'discard'):
legacy_attribute_name = '{}_these_nonmembers'.format(
action_name)> There were no list of "accept this non members" in the Postorius
config
If there is no such list in Postorius, I would guess that some developer thought "OK, this is a legacy feature and I'll get back to it when we move it to the proper database" (unfortunate, but most of us have open-subscription member-posts-only lists, and the current workflow seems to work well, so this feature may not be so salient).
I think it is solved now (deleting all and putting to hold just the non-members addresses that I would want to pass messages on)
This should not be necessary, unless there were pre-approved addresses that are no longer appropriate. But Mailman can't know that!
The fact is that any sender address of mails sent to the list is saved as a "non member", and it make me nervous as it will grow despite it is a legitimate message or not.
As far as I can see there is no other rule except that legacy rule (and the rule that accepts all posts, which is rarely used) that will accept a post from nonmembers.
Anybody knows if there is a per list config to avoid saving all that addresses as non members? o a overall config?
No, there isn't, and there never will be. This is a basic design decision about the representation of "people who participate in any way in mailing lists". The only way to prevent creation of an Address object for the sender of an email is to reject or discard the post (in which case the "create an Address" path isn't entered), and don't set a moderation action on the sender. Obviously if you set a moderation action, that will create an Address in the database.
The problem you encountered is not preventable by any means whatsoever: a malicious actor discovered an address that is allowed to post. The specific case you're worried about, that nonmembers will automatically be allowed to post, is not possible. The only way to get on the list of "accept_these_nonmembers" is to be inherited from accept_these_nonmembers on the parent Mailman 2 list, or an explicit decision by a moderator.
The fact that this list is not exposed in Postorius is a bug, but I can't say when it might be fixed. It appears that the current process (the moderator can set the sender's persistent moderation action separately from approving or rejecting a particular post) works fine, since this is the first time I've heard of this issue, and as you present it, Mailman is doing the best it can since Somebody Authorized put that sender on the approved list in Mailman 2.
If a particular nonmember becomes a problem, you can access that User or Address in Postorius and set them to Hold, Reject, or Discard.
The process of deleting all nonmembers has been more difficult than it should be, but this is being fixed, IIRC in the next release.
Steve
On 7/8/21 13:55, Stephen J. Turnbull wrote:
Guillermo Hernandez (Oldno7) via Mailman-users writes:
Yesterday I had a problem with a private list. A spammer sent a mail forging one address that was a non-member (in fact was an "accept this no subscriber mails" from mailman 2.1 import) with very bad intentions and it was distributed.
If you upgrade from Mailman 2 to Mailman 3, the accept_these_nonmembers list for each list is grandfathered into the Mailman 3 list (from rules/moderation.py):
# Check the '*_these_nonmembers' properties first. XXX These are # legacy attributes from MM2.1; their database type is 'pickle' and # they should eventually get replaced. for action_name in ('accept', 'hold', 'reject', 'discard'): legacy_attribute_name = '{}_these_nonmembers'.format( action_name)> There were no list of "accept this non members" in the Postorius
config
If there is no such list in Postorius, I would guess that some developer thought "OK, this is a legacy feature and I'll get back to it when we move it to the proper database" (unfortunate, but most of us have open-subscription member-posts-only lists, and the current workflow seems to work well, so this feature may not be so salient).
I think it is solved now (deleting all and putting to hold just the non-members addresses that I would want to pass messages on)
This should not be necessary, unless there were pre-approved addresses that are no longer appropriate. But Mailman can't know that!
The fact is that any sender address of mails sent to the list is saved as a "non member", and it make me nervous as it will grow despite it is a legitimate message or not.
As far as I can see there is no other rule except that legacy rule (and the rule that accepts all posts, which is rarely used) that will accept a post from nonmembers.
Anybody knows if there is a per list config to avoid saving all that addresses as non members? o a overall config?
No, there isn't, and there never will be. This is a basic design decision about the representation of "people who participate in any way in mailing lists". The only way to prevent creation of an Address object for the sender of an email is to reject or discard the post (in which case the "create an Address" path isn't entered), and don't set a moderation action on the sender. Obviously if you set a moderation action, that will create an Address in the database.
The problem you encountered is not preventable by any means whatsoever: a malicious actor discovered an address that is allowed to post. The specific case you're worried about, that nonmembers will automatically be allowed to post, is not possible.
Well, as I couldn't locate which legacy accepted to post non-member addresses were permitted, I solved it deleting all of them (thanks again Mark) and adding via Postorius to the list of "hold these non-members messages" that I will examine and decide if it is a legitimate send.
Thanks a lot for your detailed explanation. I can do better decisions now adjusting the configuration as needed.
The only way to get on the list of "accept_these_nonmembers" is to be inherited from accept_these_nonmembers on the parent Mailman 2 list, or an explicit decision by a moderator.
The fact that this list is not exposed in Postorius is a bug, but I can't say when it might be fixed. It appears that the current process (the moderator can set the sender's persistent moderation action separately from approving or rejecting a particular post) works fine, since this is the first time I've heard of this issue, and as you present it, Mailman is doing the best it can since Somebody Authorized put that sender on the approved list in Mailman 2.
If a particular nonmember becomes a problem, you can access that User or Address in Postorius and set them to Hold, Reject, or Discard.
The process of deleting all nonmembers has been more difficult than it should be, but this is being fixed, IIRC in the next release.
Steve
--
On 8/7/21 4:55 AM, Stephen J. Turnbull wrote:
Guillermo Hernandez (Oldno7) via Mailman-users writes:
Yesterday I had a problem with a private list. A spammer sent a mail forging one address that was a non-member (in fact was an "accept this no subscriber mails" from mailman 2.1 import) with very bad intentions and it was distributed.
If you upgrade from Mailman 2 to Mailman 3, the accept_these_nonmembers list for each list is grandfathered into the Mailman 3 list (from rules/moderation.py):
# Check the '*_these_nonmembers' properties first. XXX These are # legacy attributes from MM2.1; their database type is 'pickle' and # they should eventually get replaced. for action_name in ('accept', 'hold', 'reject', 'discard'): legacy_attribute_name = '{}_these_nonmembers'.format( action_name)> There were no list of "accept this non members" in the Postorius
config
If there is no such list in Postorius, I would guess that some developer thought "OK, this is a legacy feature and I'll get back to it when we move it to the proper database" (unfortunate, but most of us have open-subscription member-posts-only lists, and the current workflow seems to work well, so this feature may not be so salient).
Moderation in MM 3 works as follows:
Is poster address a member. If so apply the member's moderation action, else
Does poster address appear or match a pattern in the legacy (accept|hold|reject|discard)_these_nonmembers (first match in that order). If so, apply the appropriate action, else
If the poster address is not a non-member, add it as a nonmember and apply the default nonmember moderation.
I think it is solved now (deleting all and putting to hold just the non-members addresses that I would want to pass messages on)
This should not be necessary, unless there were pre-approved addresses that are no longer appropriate. But Mailman can't know that!
The import21 process imports *_these_nonmembers by adding any regexps to the corresponding MM 3 *_these_nonmembers and adding any addresses as nonmembers with the appropriate moderation action.
The fact is that any sender address of mails sent to the list is saved as a "non member", and it make me nervous as it will grow despite it is a legitimate message or not.
This is the way MM 3 works. The only reason *_these_nonmembers exists in MM 3 is to support the legacy regexps from MM 2.1. For non-regexp addresses the address is a nonmember with the appropriate moderation action.
I actually find this convenient. When a nonmember posts spam to a list and that spam is held, I discard the spam in Postorius and at the same time set the nonmember's moderation action to discard so I never see any more from that address. Also, when someone who I recognize as a list member posts from an alternate nonmember address, I can accept that post and set the nonmember address moderation action to default processing.
As far as I can see there is no other rule except that legacy rule (and the rule that accepts all posts, which is rarely used) that will accept a post from nonmembers.
As I note above, nonmembers and members alike have a moderation action which can be any of Hold, Reject (with notice), Discard (no notice), Accept (without any further checks), Default Processing (continue with additioal checks) or the list's default action for (non)members.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 7/8/21 20:03, Mark Sapiro wrote:
This is the way MM 3 works. The only reason *_these_nonmembers exists in MM 3 is to support the legacy regexps from MM 2.1. For non-regexp addresses the address is a nonmember with the appropriate moderation action.
I actually find this convenient. When a nonmember posts spam to a list and that spam is held, I discard the spam in Postorius and at the same time set the nonmember's moderation action to discard so I never see any more from that address. Also, when someone who I recognize as a list member posts from an alternate nonmember address, I can accept that post and set the nonmember address moderation action to default processing.
I find it very convenient too, but it concerns me that spammers, really "profesional" spammers and senders of phising mail change the forged address and sender domain too rapidly to track them. As of it, the non-member database is growing more and more. It could be a "monster" in two or three years...
Not a problem now. And I know now how to delete them in one go :)
--
Guillermo Hernandez (Oldno7) via Mailman-users writes:
I find it very convenient too, but it concerns me that spammers, really "profesional" spammers and senders of phising mail change the forged address and sender domain too rapidly to track them. As of it, the non-member database is growing more and more. It could be a "monster" in two or three years...
Well, let's think that out. I think we allocate a 255-byte text field for email addresses, another 255 bytes for display names, and I can't imagine that including the rest of the data fills as much as 1k, even including indicies and other database optimization overhead. So if you get a million spam messages to your lists, each from a unique address, that's only one gigabyte. To get a million spam messages in three years, you'd need to get almost a thousand per day that get through your MTA's filters to Mailman's moderation queue, which is the only place this automatic construction of nonmembers takes place. I think your moderators would let you know about that long before your disk fills up.
I'm not making fun of you, I'm explaining why I don't think this is a high priority for us, and why you don't need to worry about monster databases, either: the human resources just for looking at the queue, let alone taking action, will be a constraint long before the disk or CPU consumption for the address database is. If there's something wrong with my analysis, let me know, but as far as I can see, if the spam flood to your lists gets to that scale, the first worries are denial of service and admin exhaustion. As usual, the best answer is to stop the barbarians at the gate (ie, the MTA).
I guess if the list is configured to automatically discard all posts from non-members, we could optimize away the non-member creation. But this seems like a relatively rare case, and if there were non-local bans on addresses, it might not be a good idea anyway.
In theory, we could even arrange to do the lookup at moderation time, although we currently do it at mail receipt time. That would allow a moderator to ban the addressee on list A and then not even see the post when they go to check on list B. I suspect even professional spammers don't bother to create a unique sender address for each recipient! Obviously, this kind of service would require putting the nonmember addresses in the database, at least temporarily.
Steve
On 8/8/21 11:51, Stephen J. Turnbull wrote:
Guillermo Hernandez (Oldno7) via Mailman-users writes:
I find it very convenient too, but it concerns me that spammers, really "profesional" spammers and senders of phising mail change the forged address and sender domain too rapidly to track them. As of it, the non-member database is growing more and more. It could be a "monster" in two or three years...
Well, let's think that out. I think we allocate a 255-byte text field for email addresses, another 255 bytes for display names, and I can't imagine that including the rest of the data fills as much as 1k, even including indicies and other database optimization overhead. So if you get a million spam messages to your lists, each from a unique address, that's only one gigabyte. To get a million spam messages in three years, you'd need to get almost a thousand per day that get through your MTA's filters to Mailman's moderation queue, which is the only place this automatic construction of nonmembers takes place. I think your moderators would let you know about that long before your disk fills up.
You are absolutely right. I am the problem... well, my background of developing from the eighties of the last century, when 1k (no mistake here) meaned a difference.
I'm not making fun of you,
Never thought that.
I'm explaining why I don't think this is a high priority for us, and why you don't need to worry about monster databases, either: the human resources just for looking at the queue, let alone taking action, will be a constraint long before the disk or CPU consumption for the address database is. If there's something wrong with my analysis, let me know, but as far as I can see, if the spam flood to your lists gets to that scale, the first worries are denial of service and admin exhaustion. As usual, the best answer is to stop the barbarians at the gate (ie, the MTA).
Yes... that's the point where I expurge most of that (discarding completely email from domains that will never send a legitimate email to our lists, and in a few cases entire .XXX areas) to "normal" addresses. But I have to confess that I relayed in the mailman control of subscriptions for that in the lists. And it all went well until I made a security hole with one non-member address
I guess if the list is configured to automatically discard all posts from non-members, we could optimize away the non-member creation. But this seems like a relatively rare case, and if there were non-local bans on addresses, it might not be a good idea anyway.
And that's exactly the case of the list where a malicious mail get into: it's a private list with "discard all non subscribers addresses", but there was a legacy address imported from mailman 2 that had permission to send mails (and I was not aware)... and a bastard spammer forged it (well, it was one of many tries with different origin addresses). My fault completely.
But this is a unique case in now more than 8 months using mailman3 (and more than 12 years using mailman).
Being that list private and non publicily available, I suspect that one subscriber address book has been hijacked, and I can expect some crap from a forged subscriber address. But that is another story...
Thanks a lot, again, for all your explanations and good work.
I'm always grateful to all of you.
In theory, we could even arrange to do the lookup at moderation time, although we currently do it at mail receipt time. That would allow a moderator to ban the addressee on list A and then not even see the post when they go to check on list B. I suspect even professional spammers don't bother to create a unique sender address for each recipient! Obviously, this kind of service would require putting the nonmember addresses in the database, at least temporarily.
Steve
--
Guillermo Hernandez (Oldno7) via Mailman-users writes:
well, my background of developing from the eighties of the last century, when 1k (no mistake here) meaned a difference.
Oh, I remember those days well! It's just that I've worked on Mailman since about 2005 so have got in the habit of thinking in "server-size" units about mail. I can still be shocked, though ... when I was following DMARC develpment I was told that when Yahoo!'s contact lists were stolen, the spammers were hitting their MTAs with 1 million "recommendation from an acquaintance" spams per *minute*! I'm glad I don't have their problems! (But think about this too: it was painful, but apparently it didn't take them down! Breathtaking scale.)
I'm not making fun of you,
Never thought that.
Of course. Just being very careful, because sometimes I'm not as polite as I think I am.
Being that list private and non publicily available, I suspect that one subscriber address book has been hijacked, and I can expect some crap from a forged subscriber address. But that is another story...
Well, we still have emergency moderation of whole lists, so if necessary to protect your subscribers you can use that big hammer. Hope you don't have to, of course.
Steve
participants (3)
-
Guillermo Hernandez (Oldno7) -
Mark Sapiro -
Stephen J. Turnbull