How to store additional information about lists?
Hi together,
I need to add custom attributes to the mailing lists:
"responsible department": Free text field who is responsible from an organisational pov (not a person but an entity)
Enhance the boolean public to a tri-state: public, restricted audience and private (restricted audience means: some people could subscribe themself - how that works is a different topic but w/o the information this can't be achived) - of course this could also be implemented as a second boolean that e.g. splits "not public" into "restricted audience" and "private" (means: nothing goes wrong for code that doesn't know this difference).
Any hints how to do that best? Is it possible to store custom attributes, or should that be stored elsewhere?
Andi
Andreas Barth writes:
I need to add custom attributes to the mailing lists:
It's a rather different use case, but Systers Mailman contains extensive modifications including Systers-specific attributes. Might be a useful example (although I don't know how recently they've merged from mainline. ISTR the last time this came up the Systers repo had moved and it wasn't clear where. Try a quick Google and if no joy, I can probably put you in touch with somebody who has a clone.
Enhance the boolean public to a tri-state: public, restricted audience and private (restricted audience means: some people could subscribe themself
I'm not sure tri-state is a good idea. The boolean public controls visibility of the list in listinfo pages. I'm not sure if combining that with who-can-subscribe logic is safe, and there are pending RFEs to change the visibility logic, which might cause you annoyance at upgrade. I also am not sure it's a *bad* idea :-), although I suspect if we were to add the feature to our distribution, we'd probably implement as a per-list check for permission (this could be a list of Users, a list of Addresses, or a function to be called with one or the other).
- of course this could also be implemented as a second boolean that e.g. splits "not public" into "restricted audience" and "private" (means: nothing goes wrong for code that doesn't know this difference).
Exactly my concern.
Any hints how to do that best? Is it possible to store custom attributes, or should that be stored elsewhere?
No, I don't think there's provision for custom attributes in core. Of course you could change the database schema, but you'd have to provide a migration for that. Migrations for somebody else's custom attributes being a hard thing to automate on our side, we don't provide for them.
Does this need to be implemented in core (eg, because restricted audience users would subscribe by mail), or could you get away with a modification to Postorius only?
- Stephen J. Turnbull (turnbull.stephen.fw@u.tsukuba.ac.jp) [210522 14:33]:
Andreas Barth writes:
I need to add custom attributes to the mailing lists:
It's a rather different use case, but Systers Mailman contains extensive modifications including Systers-specific attributes. Might be a useful example (although I don't know how recently they've merged from mainline. ISTR the last time this came up the Systers repo had moved and it wasn't clear where. Try a quick Google and if no joy, I can probably put you in touch with somebody who has a clone.
Thanks, I'll give it a try.
Enhance the boolean public to a tri-state: public, restricted audience and private (restricted audience means: some people could subscribe themself
I'm not sure tri-state is a good idea. The boolean public controls visibility of the list in listinfo pages. I'm not sure if combining that with who-can-subscribe logic is safe
Perhaps I expressed it badly.
Currently it is "list public" vs "subscribers only". I need something inbetween, so that some (i.e. most) people have access to the restricted audience lists but not just everybody.
, and there are pending RFEs
to change the visibility logic, which might cause you annoyance at upgrade. I also am not sure it's a *bad* idea :-), although I suspect if we were to add the feature to our distribution, we'd probably implement as a per-list check for permission (this could be a list of Users, a list of Addresses, or a function to be called with one or the other).
That would be working for me as well.
No, I don't think there's provision for custom attributes in core. Of course you could change the database schema, but you'd have to provide a migration for that. Migrations for somebody else's custom attributes being a hard thing to automate on our side, we don't provide for them.
Just as an idea, how about always one "custom attributes" field. So that from a database pov this field is always there. In that field, the attributes could then be stored as e.g. json (which is of course a bit worse from performance than multiple entries, but well). Rest would give access to this field, but anything else is really up to the local implementation?
Does this need to be implemented in core (eg, because restricted audience users would subscribe by mail), or could you get away with a modification to Postorius only?
It could be done in postorius - the restriction that subscribing by mail needs moderators action would be ok (or if it annoys me, this part be move to a cron job).
Andi
Andreas Barth writes:
Currently it is "list public" vs "subscribers only". I need something inbetween, so that some (i.e. most) people have access to the restricted audience lists but not just everybody.
That much is clear. The problem is that "access to list" can mean a lot of things: can subscribe, can post, can admin, can view archives, can view list in listinfo, can view subscriber list, and there are probably others. What permissions are needed for what general classes of users?
Just as an idea, how about always one "custom attributes" field. So that from a database pov this field is always there.
We could do that, but then we'd have to support it. I'm thinking of situations like admin A populates the field, then leaves the organization, and we add (some of) the supported features, and admin B wants to migrate that. I don't see any good coming of that for us. As Mark says, a separate table seems like the way to go for custom attributes.
It could be done in postorius - the restriction that subscribing by mail needs moderators action would be ok (or if it annoys me, this part be move to a cron job).
OK. I'll think about this.
Steve
- Stephen J. Turnbull (turnbull.stephen.fw@u.tsukuba.ac.jp) [210523 02:41]:
Andreas Barth writes:
Currently it is "list public" vs "subscribers only". I need something inbetween, so that some (i.e. most) people have access to the restricted audience lists but not just everybody.
That much is clear. The problem is that "access to list" can mean a lot of things: can subscribe, can post, can admin, can view archives, can view list in listinfo, can view subscriber list, and there are probably others. What permissions are needed for what general classes of users?
public: anyone can subscribe and read the archive. Think about an announcement mailing list of an organisation.
restricted: for employees (or members - or whatever) only. So they could subscribe themself. The archive and posting may be restricted to subscribers or to anyone in that group - both would be working for me. (How to know who is allowed to subscribe is another question - having an internal mailing list which contains all the members just for this purpose would be ok for me.)
private: hand-selected people, think about e.g. board or steering committees internal discussions. Of course, the archive (if there is any) should be restricted to the actual members. Posting to these lists is currently not restricted, so no derivation needed from the current set in postorius.
Just as an idea, how about always one "custom attributes" field. So that from a database pov this field is always there.
We could do that, but then we'd have to support it. I'm thinking of situations like admin A populates the field, then leaves the organization, and we add (some of) the supported features, and admin B wants to migrate that. I don't see any good coming of that for us. As Mark says, a separate table seems like the way to go for custom attributes.
Well, that problem will always exist. I don't think you would make it harder by having one "custom attributes" field.
I'm going to try the plugin route Mark suggested.
Andi
Andreas Barth writes:
public: anyone can subscribe and read the archive.
This is already possible, more or less the default list configuration.
"Subscribe and read archive" involves multiple separate list attributes, enforced by three separate applications. Subscription policy is controlled by whether you need approval of an admin or not, and whether you need to verify your email or not, plus the ban lists. Archive access (assuming you have one at all) is contrlled by a different list attribute, archive_policy. Core has final say on subscriptions, but Postorius could do it if email subscriptions are disabled. HyperKitty handles the archive permissions. These various aspects are currently quite loosely coupled.
restricted: for employees (or members - or whatever) only. So they could subscribe themself. The archive and posting may be restricted to subscribers or to anyone in that group - both would be working for me.
I think this can be implemented with the controls we already have. The basic idea is to handle the "anyone in that group" case by mass subscribing and setting the subscription to no-mail. Obviously, this is most practical with a fairly static membership (eg, students at a school). A fluid membership (for example, a volunteer organization) would be made easier to manage with a "member adapter" to an external database to handle the "do I know this person" questions. ("Member adapter" is Mailman 2 terminology, I don't know if it's exactly the same in Mailman 3; it does or *should* exist in Mailman 3, though.)
It's possible that we could make this easier to configure with some changes to core. However, if this is as close to what you need as I think it is, I would advise against vendoring Mailman, and instead writing some scripts to help manage mass subscription, or create the member adapter.
(How to know who is allowed to subscribe is another question - having an internal mailing list which contains all the members just for this purpose would be ok for me.)
That's helpful to know. This sounds very similar to the Systers approach, where the Mailman core User database is the authoritative membership database for the organization.
private: hand-selected people, think about e.g. board or steering committees internal discussions. Of course, the archive (if there is any) should be restricted to the actual members. Posting to these lists is currently not restricted, so no derivation needed from the current set in postorius.
This is already possible by requiring admin approval, or by setting the ban list to something that matches everything like "^.".
On 5/25/21 12:28 AM, Stephen J. Turnbull wrote:
Andreas Barth writes:
private: hand-selected people, think about e.g. board or steering committees internal discussions. Of course, the archive (if there is any) should be restricted to the actual members. Posting to these lists is currently not restricted, so no derivation needed from the current set in postorius.
This is already possible by requiring admin approval, or by setting the ban list to something that matches everything like "^.".
The ban list can't be used in this way. A banned address can't be subscribed by any means including admin action. And even if an admin temporarily removes the ban, subscribes the address and then restores the ban, the now banned member can't post.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
- Stephen J. Turnbull (turnbull.stephen.fw@u.tsukuba.ac.jp) [210525 09:28]:
It's possible that we could make this easier to configure with some changes to core. However, if this is as close to what you need as I think it is, I would advise against vendoring Mailman, and instead writing some scripts to help manage mass subscription, or create the member adapter.
I agree, any changes to existing code in core should be mergeable.
I however think about replacing the hard dependency on allauth by an interface (which would mean minimal changes to the existing code, but would allow to plug in another class than allauth if helpful).
But that's another topic than the one I asked at the beginning of this mail :)
That's helpful to know. This sounds very similar to the Systers approach, where the Mailman core User database is the authoritative membership database for the organization.
Re Systers, I found https://bazaar.launchpad.net/~systers/systers/stable/files
This is however mm2 based. Is this what everybody hinted at, or is there also a mm3 clone I failed to find yet?
Andi
On 5/25/21 11:34 AM, Andreas Barth wrote:
Re Systers, I found https://bazaar.launchpad.net/~systers/systers/stable/files
This is however mm2 based. Is this what everybody hinted at, or is there also a mm3 clone I failed to find yet?
What you found looks like the right thing, but it appears to not be a complete Mailman but rather just the Systers added and modified files, and yes it is based on an old Mailman 2.1.
As far as I know, none of this has ever been migrated to Mailman 3.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
As far as I know, none of this has ever been migrated to Mailman 3.
It has been migrated to Mailman 3, there were at least two related GSoC projects, and the repo was on GitHub, but it's not at the same URL now. Not sure if they've stopped using it, I haven't mentored there for four or five years. Let me look around.
Steve
On 5/26/21 8:23 AM, Stephen J. Turnbull wrote:
Mark Sapiro writes:
As far as I know, none of this has ever been migrated to Mailman 3.
It has been migrated to Mailman 3, there were at least two related GSoC projects, and the repo was on GitHub, but it's not at the same URL now. Not sure if they've stopped using it, I haven't mentored there for four or five years. Let me look around.
Their lists at https://systers.org/mailman/listinfo/ are still based on Mailman 2.1.12. I have tried to find their stuff in the recent past, and all I could find were broken links.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
- Stephen J. Turnbull (turnbull.stephen.fw@u.tsukuba.ac.jp) [210526 17:24]:
Mark Sapiro writes:
As far as I know, none of this has ever been migrated to Mailman 3.
It has been migrated to Mailman 3, there were at least two related GSoC projects, and the repo was on GitHub, but it's not at the same URL now. Not sure if they've stopped using it, I haven't mentored there for four or five years. Let me look around.
https://github.com/systers/systers.github.io/wiki/Features-for-Mailman-3 seems to be existing but some links broken. https://github.com/khushboo9293/mailman/tree/Feature-CountUnsubscribers is a clone, the original code being moved or removed. However, looking through I don't see so many additions, especially not a membership adapter.
I can't find the result of GSoC 2018 which should have worked on the membership adapter.
That's so far as I can find.
Andi
Andreas Barth writes:
https://github.com/khushboo9293/mailman/tree/Feature-CountUnsubscribers is a clone, the original code being moved or removed. However, looking through I don't see so many additions, especially not a membership adapter.
There won't be a membership adapter in Systers Mailman if there isn't one in GNU Mailman, because they used the stock user database and added fields to the User record to implement their membership extensions.
Membership adapter is a Mailman 2 feature that allowed use of an external membership database (such as a corporate LDAP system). I thought it was ported to Mailman 3 but maybe not.
Steve
On 5/29/21 7:31 PM, Stephen J. Turnbull wrote:
There won't be a membership adapter in Systers Mailman if there isn't one in GNU Mailman, because they used the stock user database and added fields to the User record to implement their membership extensions.
Membership adapter is a Mailman 2 feature that allowed use of an external membership database (such as a corporate LDAP system). I thought it was ported to Mailman 3 but maybe not.
Actually, it's not quite that simple. Mailman 2.1 defines a
MemberAdaptor class, the methods of which are used for everything to do
with list members and memberships and are all documented in the base
definition in Mailman/MemberAdaptor.py and all raise
NotImplementedError. Also included in Mailman 2.1 is
Mailman/OldStyleMemberships.py which subclasses MemberAdaptor and
actually implements the methods, storing the data in the same config.pck
pickle as the list settings.
Various people have implemented various other MemberAdaptor classes for various purposes. https://bugs.launchpad.net/mailman/+bug/558106 contains several revisions of one using LDAP for the (read only) membership data.
The Mailman 3 analog of this is the IMember interface defined in mailman/interfaces/member.py and implemented in mailman/model/member.py
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro writes:
On 5/29/21 7:31 PM, Stephen J. Turnbull wrote:
Membership adapter is a Mailman 2 feature that allowed use of an external membership database (such as a corporate LDAP system). I thought it was ported to Mailman 3 but maybe not.
Actually, it's not quite that simple. Mailman 2.1 defines a MemberAdaptor class, [...]. The Mailman 3 analog of this is the IMember interface defined in mailman/interfaces/member.py and implemented in mailman/model/member.py
OK, but for Andi it is "simple" in the sense that he doesn't have an existing MemberAdapter, so the plan is "Implement the IMember interface".
Easy to say, maybe not so easy to do. ;-)
Steve
- Stephen J. Turnbull (turnbull.stephen.fw@u.tsukuba.ac.jp) [210530 09:19]:
Mark Sapiro writes:
On 5/29/21 7:31 PM, Stephen J. Turnbull wrote:
Membership adapter is a Mailman 2 feature that allowed use of an external membership database (such as a corporate LDAP system). I thought it was ported to Mailman 3 but maybe not.
Actually, it's not quite that simple. Mailman 2.1 defines a MemberAdaptor class, [...]. The Mailman 3 analog of this is the IMember interface defined in mailman/interfaces/member.py and implemented in mailman/model/member.py
OK, but for Andi it is "simple" in the sense that he doesn't have an existing MemberAdapter, so the plan is "Implement the IMember interface".
Not sure if this is the right approach, or if it wouldn't be more appropriate to "just" replace the dependency on allauth in postorius, and do all the magic in the frontend only (because that's where people actually subscribe). Whoever subscribes by mail needs some admin action, or some cron job, or is unsupported - for my case the main interface is the web interface, not the email commands.
That of course would imply that e.g. cleaning up mailing lists happens by a cron job with rest, but why not?
Andi
Andreas Barth writes:
Not sure if this is the right approach, or if it wouldn't be more appropriate to "just" replace the dependency on allauth in postorius, and do all the magic in the frontend only (because that's where people actually subscribe).
If it's purely a matter of managing subscriptions by individual users, that's the path of least resistance, I think. I would not be surprised if there are things you don't actually need, but you'd need to implement so that Postorius doesn't blow up if some IMember feature you haven't implemented gets accessed.
If in the future you might have an external personnel database with other stuff in it, and you want Mailman automatically synced to that, then IMember now might make sense to avoid technical debt in the future. But I don't know enough about IMember or your use case at the moment to make a useful recommendation.
Steve
On 5/20/21 2:28 PM, Andreas Barth wrote:
Any hints how to do that best? Is it possible to store custom attributes, or should that be stored elsewhere?
Storing custom attributes as part of the MailingList object would require modifying the model in mailman/model/mailinglist.py and probably creating an alembic migration https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/databa.... This would add significant complication to upgrading Mailman core.
I think a better approach would be to store custom attributes in a separate database table.
Also, you might look at plugins as a way to access your custom attributes via REST without having to modify Mailman core itself.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 5/22/21 8:23 AM, Mark Sapiro wrote:
Also, you might look at plugins as a way to access your custom attributes via REST without having to modify Mailman core itself.
Meant to include a link - https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/plugins/doc...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Andreas Barth wrote:
I need to add custom attributes to the mailing lists: "responsible department": Free text field who is responsible from an organisational pov (not a person but an entity) ... Any hints how to do that best? Is it possible to store custom attributes, or should that be stored elsewhere?
It's possible to do something similar with Mailman 2 - see "Bob's Unofficial Mailman Tricks & Tips" at http://nleaudio.com/bnotes/mailman.htm - I've been using it for years.
In my case it's like the Systers subscription page that Steve mentioned - something to help me decide whether to accept the subscription request, rather than something that I need to store in a database.
When I looked at Mailman 3 last year I made a short custom subscription form which included a custom field, and a script to email it to me then send an email to the listname-subscribe email address.
(I think I must have edited the MM3 confirmation link too because I've found a script that takes URL parameters, and sends a confirmation email to an address created from listname-confirm+ and the relevant token. I can't remember whether that was linked to having the custom field script.)
Best wishes
Jonathan
participants (4)
-
Andreas Barth -
jonathan.mailing.lists＠gmail.com -
Mark Sapiro -
Stephen J. Turnbull