converting bulk accept_these_nonmembers in migration from mailman 2 to 3
I have been instructed by Mark Sapiro to subscribe to this list to post questions related to mailman3.
On Mon, 31 Jan 2022, Mark Sapiro wrote:
On 1/31/22 06:20, Lucio Chiappetti wrote:
One of the lists of which I am a co-moderator has been moved from mailman 2 to 3.
Actually the migration has been made operative yesterday (and so we got the first problems ...).
Basic background. There are a number of lists within my institution (a nationwide research institution with many seats across our country), some are institutional (all staff subscribed ex officio) and managed by staff in one seat, some other are non-institutional (subscription is optional) and are managed by a few volunteers in other cities different from the seat where the mailman server runs.
In particular, the list settings for *_these_nonmembers are at Settings -> Message Acceptance, but these settings in particular are deprecated in Mailman 3 except for regexps. In Mailman 3 the preferred way to say accept a nonmember is to create a nonmember for that user and set it's Moderation action to Default Processing. If, a nonmember posts, the nonmember record is automatically created,
Let me explain our case, and how it was managed in mailman2.
originally our staff (of the order of 1000) had addresses like user@place.inaf.it or user@place.astro.it. Some had even two (say place1.inaf.it and place2.inaf.it), so they subscribed with ONE of their work addresses
later we got addresses like name.surname@inaf.it. New users subscribe with this address but MANY of the old ones continue to use the PREVIOUS address.
we implemented a spam filtering policy as follows:
staff (and former staff) can SUBSCRIBE with any address or CHANGE their address. Any subscribed address will be recognised. Mail is NOT held in moderation unless too bulky, or has undisclosed recipients
mail from non members with any address in the .it TLD or in the tng.iac.es domain (where some of our staff work) goes into moderation (this is achieved via regexp in hold_these_nonmembers)
anything else is silently discarded as spam (generic_nonmember_action = discard)
when some staff posts from a non-subscribed address (be it the home address, or one of the alternate addresses described above) it therefore goes into moderation because of hold_these_nonmembers
in mailman2 we could let them pass "manually" (if we recognised the person) or (in case they used an alternate address) tick to add them permanently to accept_these_nonmembers
(in one tick)
at the end we got somehow fed up of this but we did NOT want to accept any address of the form ^.+@.+\.inaf.it (which could be forged) but found that we could simply put in accept_these_nonmembers two entries like @list1 @list2
where list1 and list2 are two institutional lists on the same server which contain all permanent and temporary staff or assimilated with their official address (these lists are automagically generated each week)
(so at least any VALID EXISTING official address is accepted)
What I found now (after the migration done by the administrators of the central seat) is that in mailman3:
accept_these_nonmembers is empty
there are 189 entries listed in Non-members (which look those formerly in accept_these_nonmembers)
hold_these_nonmembers was copied identical with the same regexp's as in mailman2
I have inspected a few of them and found that they have Administration options Moderation set to List default
the effect of this apparently was that a post from some.guy@inaf.it (whose subscription address is guy@somewhere.inaf.it) was hold for moderation
is this due to a loop with the regexp in hold_these_nonmembers ?
the default action would be to let it pass
Or should we set Administration options Moderation to Accept immediately ?
if so, how can we do it automatically for all 189 entries ?
or move back the 189 addresses to accept_these_nonmembers ?
or for the future have persons posting with alternate addrresses accepted for the future WITH ONE CLICk ?
AND LAST BUT NOT LEAST would the syntax @list1 in accept_these_nonmembers work in mailman3 ?
already putting it back will save a lot of work.
Many thanks
-- Lucio Chiappetti - INAF/IASF - via Corti 12 - I-20133 Milano (Italy) For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
"All that is google does not glitter Nor all who use alpine/procmail are lost"
On 2/8/22 09:01, Lucio Chiappetti wrote:
Snipping to what I thing are the issues.
What I found now (after the migration done by the administrators of the central seat) is that in mailman3:
- accept_these_nonmembers is empty
- there are 189 entries listed in Non-members (which look those formerly in accept_these_nonmembers)
And those non-members all have there moderation action set to Defer which means their posts will be accepted but the additional checks such as too big, etc. will still be applied.
- hold_these_nonmembers was copied identical with the same regexp's as in mailman2
This is all expected. MM 3, unlike MM 2.1 has a concept of nonmembers with moderation action, and this is intended to replace *_these_nonmembers.
The legacy *_these nonmembers attributes still exist to support regexps. This is why the addresses were made nonmembers with the appropriate moderation action, but the regexps were retained.
- I have inspected a few of them and found that they have Administration options Moderation set to List default
I'm not sure what you are looking at here. Imported nonmembers should all have moderation action set according to which *_these_nonmembers they came from. Imported members should have their moderation action set to Defer if they were unmoderated and if they were moderated, it should be set based on the 2.1 list's member_moderation_action.
- the effect of this apparently was that a post from some.guy@inaf.it (whose subscription address is guy@somewhere.inaf.it) was hold for moderation
is this due to a loop with the regexp in hold_these_nonmembers ?
It is because all the legacy *_these_nonmember actions are applied before nonmember moderation checks, so if an address matches a regex in hold_these_nonmembers, the post will be held regardless of the nonmember's moderation action. This may actually be a bug. I'll have to think about that. I've filed https://gitlab.com/mailman/mailman/-/issues/978 on this.
the default action would be to let it pass
Or should we set Administration options Moderation to Accept immediately ?
No. You can avoid this issue by adding the 189 nonmember addresses to accept_these_nonmembers (as addresses, not regexps). Addresses still work, and accept_these_nonmembers takes precedence over hold_these_nonmembers just as in MM 2.1.
- if so, how can we do it automatically for all 189 entries ?
- or move back the 189 addresses to accept_these_nonmembers ?
Yes, move them back.
- or for the future have persons posting with alternate addrresses accepted for the future WITH ONE CLICk ?
If they are nonmembers, you can accept the post and set moderation action in one operation in Postorius, but the regexps in hold_these_nonmembers will still take precedence.
- AND LAST BUT NOT LEAST would the syntax @list1 in accept_these_nonmembers work in mailman3 ?
This is https://gitlab.com/mailman/mailman/-/issues/794 (still open)
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Dear Mark, thanks for your interest.
On Fri, 11 Feb 2022, Mark Sapiro wrote:
This may actually be a bug. I'll have to think about that. I've filed https://gitlab.com/mailman/mailman/-/issues/978 on this.
This is https://gitlab.com/mailman/mailman/-/issues/794 (still open)
Given the time needed to solve the issues (I will anyhow notify the site administrators, hopting they would update mailman when/if resolved), this means we have to find a workaround.
And those non-members all have there moderation action set to Defer which means their posts will be accepted but the additional checks such as too big, etc. will still be applied.
not sure what you mean by "defer". The non-member options I see are called
List default -- follow the list's default member action. Hold -- This holds the message for approval by the list moderators. Reject -- this automatically rejects the message by sending a bounce notice to the post's author. The text of the bounce notice can be configured by you. Discard -- this simply discards the message, with no notice sent to the post's author. Accept -- accepts any postings without any further checks. Default Processing -- run additional checks and accept the message.
If they are nonmembers, you can accept the post and set moderation action in one operation in Postorius, but the regexps in hold_these_nonmembers will still take precedence.
OK, I've found where to do it, it requires to select the held message and scroll to the bottom then set Moderation from a menu.
But if hold_these_nonmembers takes priority this will be uselesws in the current setting.
I'm not sure what you are looking at here. Imported nonmembers should all have moderation action set according to which *_these_nonmembers they came from. Imported members should have their moderation action set to Defer if they were unmoderated and if they were moderated, it should be set based on the 2.1 list's member_moderation_action.
I go to the non-member list, select non-member options and at the bottom see an item Administration options Moderation. The possible values are those I pasted above.
The actual value for most of the imported non-members is "Default processing" while it is "List default" for thoase automatically added afresh, It is "Discard" for a number of imported non-members with spam-looking addresses (I guess they were in some other *_not_members part of the standard 2.1 antispam ... I can't recall me doing something on those)
Side question: is there a way to operate IN BULK on non-members ? Like it is for members (I exported them to CSV).
- if so, how can we do it automatically for all 189 entries ?
- or move back the 189 addresses to accept_these_nonmembers ?
Yes, move them back.
I am not so concerned of these 189 (or better of the 189 minus the discard ones which should stay) ... anyhow moving them back will be uncomfortable if there is not a bulk operation tool.
I am more concerned of the new cases automatically added.
Our member list includes 1043 addresses. Of these less than 20 use an external address (say gmail), so they post "from home" and will pass. 194 addresses are of the form name.surname@inaf.it (these are likely to be recently hired staff, who have and use onlyu the organization-wide address).
All the rest (so still more than 800, can't really ask all of them to re-register (*)) are of the form username@institute.inaf.it (and it won't be easy also to bulk-delete and bulk-resubscribe because of the inhomogeneous mapping of username to name.surname.
Of these about 800 some are lurkers and are no problem. Some are frequent posters and remember to post from username@institute.inaf.it and are no problem. Other may post from an alternate address (maybe even the old address exists as a receiving-only alias), and in this case with the current arrangement they will bo on hold every time.
I guess we should wait for the solution of issue 794
(*) by the way do you confirm that if one is subscribed to a list with an address (be it bulk subscription, import or new fast subscription), this person CANNOT change their settings (or address) unless one does a "sign in" to a (Postorius) account ?
It is because all the legacy *_these_nonmember actions are applied before nonmember moderation checks, so if an address matches a regex in hold_these_nonmembers, the post will be held regardless of the nonmember's moderation action.
Addresses still work, and accept_these_nonmembers takes precedence over hold_these_nonmembers just as in MM 2.1.
This was not my impression when I ran a thorough test of 12 cases. Except the obvious case of the subscriber posting from the subscription address, all messages went on hold.
Whatever the non-member moderation would be (even if set to reject ! ... but that makes sense if *_these_nonmembers prevail on non-member options), and even if the address was set explicitly in accept_these_nonmembers (should I delete it from the non-member list ?)
What is the actual order (or flow chart) of choices ?
*_these_nonmembers in the order listed in the page (within each in the order of occurrence of regexp's) then non-member options ?
If I have Name.Surname@inaf.it in accept_these_nonmembers, then hold_these_nonmembers set to
^.+@.+\.it ^.+@.+\.tng\.iac\.es
what do you foresee ?
And for different regexps, will processing terminate at the first non-matching one, will they be ANDed or ORed, or what ?
I am reluctant to accept inconditionally ^.+\..+@inaf\.it (they may be falsified, only the check on their presence in the other list (issue 794)
I guess I should remove the ^.+@.+\.it regexp AND AT THE SAME TIME change the Default action to take when a non-member posts to the list to Hold instead of Discard.
This way we'll have to check spam (instead of having it auto-discarded), but if people with alternate addresses post, once they are entered in the Non-member list they can be authorized to pass for the future ...
.. would it work ?
-- Lucio Chiappetti - INAF/IASF - via Corti 12 - I-20133 Milano (Italy) For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
On 2/12/22 08:19, Lucio Chiappetti wrote:
Dear Mark, thanks for your interest.
On Fri, 11 Feb 2022, Mark Sapiro wrote:
This may actually be a bug. I'll have to think about that. I've filed https://gitlab.com/mailman/mailman/-/issues/978 on this.
This is https://gitlab.com/mailman/mailman/-/issues/794 (still open)
Given the time needed to solve the issues (I will anyhow notify the site administrators, hopting they would update mailman when/if resolved), this means we have to find a workaround.
https://gitlab.com/mailman/mailman/-/issues/978 has been fixed. You can apply the fix to mailman/rules/moderation.py from https://gitlab.com/mailman/mailman/-/merge_requests/961/diffs
not sure what you mean by "defer". The non-member options I see are called
List default -- follow the list's default member action. Hold -- This holds the message for approval by the list moderators. Reject -- this automatically rejects the message by sending a bounce notice to the post's author. The text of the bounce notice can be configured by you. Discard -- this simply discards the message, with no notice sent to the post's author. Accept -- accepts any postings without any further checks. Default Processing -- run additional checks and accept the message.
defer is what's called Default Processing in the above list.
But if hold_these_nonmembers takes priority this will be uselesws in the current setting.
True, but if you apply the fix for https://gitlab.com/mailman/mailman/-/issues/978, hold_these_nonmembers won't take priority.
I go to the non-member list, select non-member options and at the bottom see an item Administration options Moderation. The possible values are those I pasted above.
The actual value for most of the imported non-members is "Default processing" while it is "List default" for thoase automatically added afresh, It is "Discard" for a number of imported non-members with spam-looking addresses (I guess they were in some other *_not_members part of the standard 2.1 antispam ... I can't recall me doing something on those)
That all seems exactly what it should be.
Side question: is there a way to operate IN BULK on non-members ? Like it is for members (I exported them to CSV).
No. Postorius doesn't currently support a CSV export for nonmembers as
it does for members, but you could do the equivalent in mailman shell
$ mailman shell -l list.example.com
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
The variable 'm' is the list.example.com mailing list
>>> for nonmemb in m.nonmembers.members:
... print(nonmemb.address.email)
...
I am not so concerned of these 189 (or better of the 189 minus the discard ones which should stay) ... anyhow moving them back will be uncomfortable if there is not a bulk operation tool.
I am more concerned of the new cases automatically added.
Apply the patch.
...
(*) by the way do you confirm that if one is subscribed to a list with an address (be it bulk subscription, import or new fast subscription), this person CANNOT change their settings (or address) unless one does a "sign in" to a (Postorius) account ?
That is correct. User's can add multiple addresses to their identity, change addresses and change other settings, but currently that has to be done via Postorius or another web UI that communicates with Mailman Core's REST API.
...>> Addresses still work, and accept_these_nonmembers takes precedence
over hold_these_nonmembers just as in MM 2.1.
This was not my impression when I ran a thorough test of 12 cases. Except the obvious case of the subscriber posting from the subscription address, all messages went on hold.
If a nonmember address is in accept_these_nonmembers, a post from that address should be accepted regardless of hold_these_nonmembers.
Whatever the non-member moderation would be (even if set to reject ! ... but that makes sense if *_these_nonmembers prevail on non-member options), and even if the address was set explicitly in accept_these_nonmembers (should I delete it from the non-member list ?)
You can leave it as a nonmember.
What is the actual order (or flow chart) of choices ?
Without the fix for https://gitlab.com/mailman/mailman/-/issues/978 the order is
- Is the sender a member? If so, apply the member's moderation action or list default.
- The sender is a nonmember, if the address matches one of (accept|hold|reject|discard)_these_nonmembers (checked in that order), apply that action.
- If no match above, apply the nonmember's moderation action - will be
List Defaultfor new nonmembers.
With the fix for https://gitlab.com/mailman/mailman/-/issues/978 the order is
- Is the sender a member? If so, apply the member's moderation action or list default.
- The sender is a nonmember, if the nonmember's moderation action is other than List Default, apply that action.
- If not, if the address matches one of (accept|hold|reject|discard)_these_nonmembers (checked in that order), apply that action.
- If none of the above, apply the list's default nonmember action.
*_these_nonmembers in the order listed in the page (within each in the order of occurrence of regexp's) then non-member options ?
That's the way it works without the fix except the order of regexps/addresses in a *_these_nonmembers list is immaterial. if the address matches any one of them, that action applies.
If I have Name.Surname@inaf.it in accept_these_nonmembers, then hold_these_nonmembers set to
^.+@.+\.it ^.+@.+\.tng\.iac\.es
what do you foresee ?
Assuming Name.Surname@inaf.it is not a member, a post from Name.Surname@inaf.it will not be held for nonmember moderation.
And for different regexps, will processing terminate at the first non-matching one, will they be ANDed or ORed, or what ?
They will all be checked until one matches or none do., I.e., effectively ORed.
I am reluctant to accept inconditionally ^.+\..+@inaf\.it (they may be falsified, only the check on their presence in the other list (issue 794)
I guess I should remove the ^.+@.+\.it regexp AND AT THE SAME TIME change the Default action to take when a non-member posts to the list to Hold instead of Discard.
This way we'll have to check spam (instead of having it auto-discarded), but if people with alternate addresses post, once they are entered in the Non-member list they can be authorized to pass for the future ...
.. would it work ?
I think that would work, but if you just apply the patch from https://gitlab.com/mailman/mailman/-/merge_requests/961/diffs, it will work the way it did in MM 2.1 honoring a nonmember's explicit moderation action before checking *_these_nonmembers.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (2)
-
Lucio Chiappetti -
Mark Sapiro