Deactivate list without deleting it?
Hi,
I'm experimenting with a port of Launchpad's mailing list infrastructure to Mailman 3. If Barry's reading, he may remember that this is no trivial task, so I'm very definitely not promising that it will be done any time soon. :-) However, for the most part it looks relatively tractable with the aid of the plugin improvements in Mailman 3.2.0; I have the rough outline of a plugin that implements much the same feature set as our current pile of monkey-patches against Mailman 2.1 in a much nicer way.
One of the major things I haven't figured out yet is list deactivation and reactivation. In our current 2.1-based code, you can deactivate a list in a reversible way, which basically tars up the list directory so that the list archives can be restored later. This isn't a native operation in 2.1, but it could be done because everything lived on the filesystem. In 3, it's less trivial. We could go to some effort to dump things out of the HyperKitty database and restore it on reactivation, but that seems rather complex and a bit silly.
Ideally, I'd like mailing lists to have a "deactivated" state in addition to full deletion. In this state, the list would appear nonexistent for most purposes: incoming email would be rejected, Postorius would deny its existence, etc. There'd need to be corresponding extensions to the IArchiver interface which caused the archiver to put its archives of that list into a corresponding deactivated state, in which it would retain the messages but queries for them would return 404 or similar; these extensions would need to be optional, since while it should be relatively easy to do this in HyperKitty it could be hard or impossible for other archivers, and perhaps this should be a separate operation from deactivating the list itself since people may want to close a list while leaving its archives accessible.
Has there been any discussion of this in the past, or any advice as to implementation strategy?
Thanks,
-- Colin Watson [cjwatson@ubuntu.com]
Colin Watson writes:
Ideally, I'd like mailing lists to have a "deactivated" state in addition to full deletion. In this state, the list would appear nonexistent for most purposes: incoming email would be rejected, Postorius would deny its existence, etc.
[...]
Has there been any discussion of this in the past, or any advice as to implementation strategy?
I don't recall anybody discussing this. Your use case is your use case, but I think if we were going to implement this I'd probably want "partial" deactivation where the archives remain available but no mail is delivered, can't subscribe, etc. I'm also not sure I'd want Postorius to deny it ever existed. That seems rather unfriendly (again, I don't understand your use case, so that is not a comment on your requirements). So to get it into the distribution we'd need to make these things optional. (Perhaps if these options proliferate much, they could be incorporated into list "styles" so that instead of setting all the options manually, the primary use cases could be served by "intuitively" named styles.)
It seems to me that dump, delete, and restore may be the best approach for the HyperKitty side because HyperKitty maintains all that state. Of course this could be done inside the database. Also, perhaps you can (ab)use the ArchivePolicy enum by adding an "inactive" element, but core has no way to enforce those on the archiver (which can be a third party host like mail-archive.com), and I'm not sure HyperKitty ever consults core about that once the archive is created.
I'm not sure what you mean about IArchiver; as far as I can see that interface is sort of half-duplex: it provides the information core needs *about* the archiver to do core's side of archiving, not for core to communicate *with* the archiver. In fact all of the concrete code is callbacks provided by the archiver, and once a message is archived the core forgets about it. So this isn't "just" an extension; it's a fundamental redesign of IArchiver.
I'm not sure that's not a good thing, but we do need to think about it. ISTM we're gradually converging back on the monolithic approach to MLM, with the distribution, administration, and archiving quite tightly interwoven, and I'm not sure I like that. OTOH, we do get a lot of requests for ever more bundling.
On Tue, Jan 16, 2018 at 10:59:28AM +0900, Stephen J. Turnbull wrote:
Colin Watson writes:
Ideally, I'd like mailing lists to have a "deactivated" state in addition to full deletion. In this state, the list would appear nonexistent for most purposes: incoming email would be rejected, Postorius would deny its existence, etc.
[...]
Has there been any discussion of this in the past, or any advice as to implementation strategy?
I don't recall anybody discussing this. Your use case is your use case, but I think if we were going to implement this I'd probably want "partial" deactivation where the archives remain available but no mail is delivered, can't subscribe, etc. I'm also not sure I'd want Postorius to deny it ever existed. That seems rather unfriendly (again, I don't understand your use case, so that is not a comment on your requirements).
I possibly should have explained our situation in a little more detail. The way we have things set up with Mailman 2.1 is that any team in Launchpad (teams are basically just a collection of people, with varying policies on who can join them etc.) can have a mailing list if its administrators want one, and they enable it in the Launchpad web UI. If you want to subscribe to one of these lists, you join the team in question and you press the "Subscribe to mailing list" button. Launchpad runs the show and tells Mailman exactly what to do. So in fact we have all of Mailman's user interface disabled except for the list archives, and Postorius wouldn't come into it in our case - I probably shouldn't have confused things by bringing it up hypothetically.
We'd probably want to do roughly the same kind of thing with Mailman 3, at least to start with, though we may well want to make some of HyperKitty's advanced features work with single-sign-on integration. I haven't looked into that in any detail yet.
I can definitely see that both "preserve archives" vs. "hide archives" are potentially useful things to do.
It seems to me that dump, delete, and restore may be the best approach for the HyperKitty side because HyperKitty maintains all that state. Of course this could be done inside the database. Also, perhaps you can (ab)use the ArchivePolicy enum by adding an "inactive" element, but core has no way to enforce those on the archiver (which can be a third party host like mail-archive.com), and I'm not sure HyperKitty ever consults core about that once the archive is created.
I'd been thinking more of a new enum in HyperKitty for the visibility state, since it does seem like quite an archiver-specific thing. (Dump/delete/restore would of course be possible, but it's somewhat cumbersome to do externally and it seems like an odd way to go about things internally given the ability to change the DB schema.) I agree that not all archiver implementations can support this.
I'm not sure what you mean about IArchiver; as far as I can see that interface is sort of half-duplex: it provides the information core needs *about* the archiver to do core's side of archiving, not for core to communicate *with* the archiver. In fact all of the concrete code is callbacks provided by the archiver, and once a message is archived the core forgets about it. So this isn't "just" an extension; it's a fundamental redesign of IArchiver.
Well, IArchiver.archive_message sends a message from core to the archiver. I'd been thinking of something like IArchiver.set_list_visibility(mlist, visibility) which would send a message from core to the archiver to instruct it to show/hide the archives. It doesn't seem too out of line, though I'm not wedded to it either.
I suppose that one alternative could be that we handle this in a Launchpad-specific add-on to HyperKitty, since at a minimum we'll need some of our own code there to deal with sign-in for private lists. So the same bit of custom code that determines whether a list's archives are private could hide them from everyone except their owner, or even deny their existence entirely. It sounds like maybe we'd be better off pursuing that?
I'm not sure that's not a good thing, but we do need to think about it. ISTM we're gradually converging back on the monolithic approach to MLM, with the distribution, administration, and archiving quite tightly interwoven, and I'm not sure I like that. OTOH, we do get a lot of requests for ever more bundling.
Given our requirements, the decreased coupling in Mailman 3 is a very good thing, and I'm definitely not looking for more bundling. I hope I've clarified that above.
Thanks,
-- Colin Watson [cjwatson@ubuntu.com]
On Jan 16, 2018, at 02:51, Colin Watson <cjwatson@ubuntu.com> wrote:
I possibly should have explained our situation in a little more detail. The way we have things set up with Mailman 2.1 is that any team in Launchpad (teams are basically just a collection of people, with varying policies on who can join them etc.) can have a mailing list if its administrators want one, and they enable it in the Launchpad web UI. If you want to subscribe to one of these lists, you join the team in question and you press the "Subscribe to mailing list" button.
Oh yeah, that was another thing that Launchpad inspired: the whole preferred email address thing. Thinking about it again now, I’m actually still kind of amazed that we could fake a one-unified-account-per-user facade over Mailman 2.1’s each-list-is-a-silo view of the world. At least you won’t have *that* mess to worry about any more. :)
I'd been thinking more of a new enum in HyperKitty for the visibility state, since it does seem like quite an archiver-specific thing.
Yep, I think that’s probably the way to do it. Then the question is who and how is HK informed about these state changes?
Well, IArchiver.archive_message sends a message from core to the archiver. I'd been thinking of something like IArchiver.set_list_visibility(mlist, visibility) which would send a message from core to the archiver to instruct it to show/hide the archives. It doesn't seem too out of line, though I'm not wedded to it either.
I don’t think that’s the right way to do it, because as Steve points out, IArchiver is really a half-duplex interface. It’s there solely to abstract away the different mechanisms for getting messages from MM3 to the archivers of choice. Even the API for getting a permalink is generally not supposed to function by direct 2-way communication with the archiver, although if an implementation of IArchiver wanted to do that, we can’t stop it.
I suppose that one alternative could be that we handle this in a Launchpad-specific add-on to HyperKitty, since at a minimum we'll need some of our own code there to deal with sign-in for private lists. So the same bit of custom code that determines whether a list's archives are private could hide them from everyone except their owner, or even deny their existence entirely. It sounds like maybe we'd be better off pursuing that?
Yes, I think so. The other way seems too round about; you’d make a REST call from LP to Core, then Core would call a method in an interface whose implementation then sends a message to HK. Having LP talk directly to HK is much more straightforward, and also gives you the ability to have HK respond with various states or information as needed. The other way, there’d be no good way to get information back up from HK to LP.
Hope that helps, and best of luck! I think it would be a fun project… if I was retired or it was 5 years ago. :) Please do come back with any further questions.
-Barry
Colin Watson writes:
I possibly should have explained our situation in a little more detail.
Well, I was pretty clear on your requirements, it turns out. I don't, and I believe I'm in agreement with the rest of the team here, need to know why you want something (at least not at this "let's see if we can hash out the requirements and the design a bit" stage). After all, even if we end up WONTFIXing it, it's open source and we're happy to talk about how you might do something even if we won't do it for you!
Postorius wouldn't come into it in our case
That's an interesting detail. Thing is, at present Postorius *is* the "control panel" for the whole suite (all the best bands are run by their bassists, you know), so we would bring it up if you left it out.
However, if you don't want to use Postorius, I think the mailman-client functionality will make it strightforward to implement your UI within Launchpad, and it now includes a nicer CLI so if you prefer to implement core and Hyperkitty functionality before the Launchpad side, mailman-client will allow you to test core features at that stage.
I'd been thinking more of a new enum in HyperKitty for the visibility state, since it does seem like quite an archiver-specific thing.
Well, there is an enum in core for it already. I'm not sure how that works currently (specifically, how you go about telling Hyperkitty if that changes). Keeping that in sync is an issue (at least for us), since Postorius can query it, IIRC.
For the rest, I have nothing to add to what Barry already wrote.
Steve
On Wed, Jan 17, 2018 at 04:34:48PM +0900, Stephen J. Turnbull wrote:
Colin Watson writes:
I possibly should have explained our situation in a little more detail.
Well, I was pretty clear on your requirements, it turns out. I don't, and I believe I'm in agreement with the rest of the team here, need to know why you want something (at least not at this "let's see if we can hash out the requirements and the design a bit" stage). After all, even if we end up WONTFIXing it, it's open source and we're happy to talk about how you might do something even if we won't do it for you!
I appreciate that approach. At the same time, I also asked because I suspected I didn't quite understand the design principles involved. Thanks to both you and Barry for clarifying the intent of IArchiver; I'll drop that idea.
Postorius wouldn't come into it in our case
That's an interesting detail. Thing is, at present Postorius *is* the "control panel" for the whole suite (all the best bands are run by their bassists, you know), so we would bring it up if you left it out.
I'm not ruling out using it in future, but right now I think we'd prefer to stick with the approach of having the "control panel" integrated tightly into LP.
However, if you don't want to use Postorius, I think the mailman-client functionality will make it strightforward to implement your UI within Launchpad, and it now includes a nicer CLI so if you prefer to implement core and Hyperkitty functionality before the Launchpad side, mailman-client will allow you to test core features at that stage.
Indeed. And I'd managed to miss the existence of mailmanclient, so thanks for bringing that up.
I'd been thinking more of a new enum in HyperKitty for the visibility state, since it does seem like quite an archiver-specific thing.
Well, there is an enum in core for it already. I'm not sure how that works currently (specifically, how you go about telling Hyperkitty if that changes). Keeping that in sync is an issue (at least for us), since Postorius can query it, IIRC.
I just looked into what HK does, and I notice that it syncs the archive policy from Core as part of a regular update job. So I think the answer is that we just use that: perhaps set the ArchivePolicy to private and arrange that nobody has the ability to view it other than maybe list owners, or possibly add a new item to ArchivePolicy that's something like "deactivated" or "hidden". At any rate that looks a lot more tractable since most of the mechanism already exists.
Thanks,
-- Colin Watson [cjwatson@ubuntu.com]
On Jan 14, 2018, at 09:45, Colin Watson <cjwatson@ubuntu.com> wrote:
I'm experimenting with a port of Launchpad's mailing list infrastructure to Mailman 3. If Barry's reading, he may remember that this is no trivial task, so I'm very definitely not promising that it will be done any time soon. :-)
For those of you not up on your history, this is interesting for several reasons. I was originally hired by Canonical 11 years ago to integrate Mailman with Launchpad. At the time Launchpad was not free software, so a lot of the design of the integration had to work within those constraints (LP was released under the AGPL in 2009 IIRC). At the time it was painful, especially because it had to keep two completely databases in sync in both directions — not an easy task! Add to that the fact that Mailman 2.1 doesn’t really have a database, just a bunch of pickle files. :)
I’m proud that the mailing list feature of Launchpad eventually became pretty darn solid, with much credit to Colin, Curtis Hovey, and many of my other wonderful former colleagues on the Launchpad team and at Canonical.
This experience also heavily influenced my thinking about the design for Mailman 3, and led directly to the REST-based API. Launchpad has a REST API, spearheaded by Leonard Richardson, who literally wrote the O’Reilly books on web services. And while there were some compromised on “pure” REST in Launchpad, the discussions we had at that time committed my thinking to as pure an approach as I was capable of doing.
However, for the most part it looks relatively tractable with the aid of the plugin improvements in Mailman 3.2.0; I have the rough outline of a plugin that implements much the same feature set as our current pile of monkey-patches against Mailman 2.1 in a much nicer way.
Very cool!
You’ve provided some clarification in a follow up, and I’ll respond there. Apologies for being a little disjointed.
One of the things that may really help you, but which doesn’t exist in a fully baked way in Core is the ability to set web hooks for various events. Core does have an event notification system (via zope.events) and that’s used to inform other parts of the Core about things like subscriptions/unsubscriptions, etc. Where web hooks would be useful is if some event in Core needs to inform Launchpad of that event. However, it’s possible that this falls under the plugins you’ve thought about - and while not general purpose, I’m positive you could do the same kind of MM3->LP notifications that you might find necessary, in custom plugins.
Ideally, I'd like mailing lists to have a "deactivated" state in addition to full deletion. In this state, the list would appear nonexistent for most purposes: incoming email would be rejected,
I forget whether LP uses Exim or Postfix. At least for the latter, it would be fairly easy (if involving a database migration) to include an “active” or “enabled” flag which would be consulted when regenerating the Postfix transport files. I would add this to the MailingList model and make it available through the REST API. Then the LP team admin can toggle this on and off as desired.
I also wouldn’t use the IArchiver interface for disabling HyperKitty access to the archives, but I’ll respond in more detail in the next follow up.
Postorius would deny its existence, etc.
You’ve clarified that Postorius isn’t needed for Launchpad, since it provides what effectively takes it places in the simplified administration u/i.
Cheers, -Barry
[Sorry for having dropped this thread on the floor for a while.]
On Tue, Jan 16, 2018 at 09:32:25PM -0800, Barry Warsaw wrote:
I'm experimenting with a port of Launchpad's mailing list infrastructure to Mailman 3. If Barry's reading, he may remember that this is no trivial task, so I'm very definitely not promising that it will be done any time soon. :-) [...] I’m proud that the mailing list feature of Launchpad eventually became
On Jan 14, 2018, at 09:45, Colin Watson <cjwatson@ubuntu.com> wrote: pretty darn solid, with much credit to Colin, Curtis Hovey, and many of my other wonderful former colleagues on the Launchpad team and at Canonical.
Shoulders of giants and all that. :-)
However, for the most part it looks relatively tractable with the aid of the plugin improvements in Mailman 3.2.0; I have the rough outline of a plugin that implements much the same feature set as our current pile of monkey-patches against Mailman 2.1 in a much nicer way.
Very cool!
You’ve provided some clarification in a follow up, and I’ll respond there. Apologies for being a little disjointed.
One of the things that may really help you, but which doesn’t exist in a fully baked way in Core is the ability to set web hooks for various events. Core does have an event notification system (via zope.events) and that’s used to inform other parts of the Core about things like subscriptions/unsubscriptions, etc. Where web hooks would be useful is if some event in Core needs to inform Launchpad of that event.
Web hooks might be interesting to look at (if they were fully baked, of course), but I'd have some reservations. The problem with that approach is that it tends to be "fire and forget", and in particular if one part of the system is down or temporarily unreachable or whatever then state can get out of sync. I think it would be unfortunate to end up with a system where a user could end up inconsistently subscribed in multiple databases due to a temporary network partition, and we might never notice.
IME web hooks work better in situations where their state is visible and it's easy for somebody to redispatch them if they seem to have got lost. GitHub's integration with various third-party services like Travis is a good example.
For this reason, I usually favour "catch-up" approaches where one service asks another for all the events that have happened since a particular date (if it isn't just asking the other service for information synchronously, of course). I suppose this is what people call "eventually consistent". And in any case I think in our system events are generally going to flow from Launchpad to Mailman rather than the other way round. So, rather than notifications, I'd been planning to stick with something broadly along the lines of how Launchpad's Mailman 2 integration works: we have a runner in the Mailman plugin that talks to a Launchpad API that publishes a sequence of pending actions, and consumes those to keep itself up to date. It may be a bit clunky, but it's essentially robust.
Alternatively, we could schedule jobs in Launchpad's own job queuing system any time a relevant event (new list, subscription, etc.) happens, and have those talk to Mailman's REST API with a backoff-and-retry arrangement. Maybe that'd be easier.
Ideally, I'd like mailing lists to have a "deactivated" state in addition to full deletion. In this state, the list would appear nonexistent for most purposes: incoming email would be rejected,
I forget whether LP uses Exim or Postfix.
I ... am actually not 100% sure. But a Mailman 3 migration would involve a complete redeployment of the mailing list components on cloud instances anyway, so we're not bound to what we have at the moment.
At least for the latter, it would be fairly easy (if involving a database migration) to include an “active” or “enabled” flag which would be consulted when regenerating the Postfix transport files. I would add this to the MailingList model and make it available through the REST API. Then the LP team admin can toggle this on and off as desired.
Yes, something like that. Though maybe we could just change its posting chain to 'reject' or similar with a "this list is closed" message instead, and dispense with the need for an explicit enabled flag?
-- Colin Watson [cjwatson@ubuntu.com]
participants (3)
-
Barry Warsaw -
Colin Watson -
Stephen J. Turnbull