Mailman 3 Status Update Please
I have been following the progress and difficulties with great interest
as many struggle to implement the new version of the great Mailman 3
list-server software. Last year about this time I posed some questions
regarding the current development status and recognize that many
additional problems seem to get in the way of actually working on long
term big picture issues.
From my perspective of being an senior (ie overaged) person desirous of implementing fully-featured, capable, scalable, reliable, secure applications such as Mailman, I am convinced that I am thoroughly over my head.
From my limited perspective, it would seem to me that a 'nice to have' system would incorporate the following.
- An open container standard configurable system install, compatible with different web & mail servers, DB utilities, etc;
- Python 3 version Django utilities for archiving and mail list name management;
- A design/plan from an inside expert perspective stating costs, time and resources to implement the plan, incorporating crowdfunding to pay for project.
- As a backup incentive to the crowdfunding plan might be the offer of an expert to set up a system for each significant amount donated, say US$1,000, such as might be accomplished by an SSL session on the target computer/server after the user has specified all necessary detail as required in a template.
I am sure that the program author and other major contributors could greatly enhance this list.
In summary, I guess what I am requesting is an overview forecast by half year period and out for the next two years?
Please share the big picture?
Thank you in advance and best regards.
Dave Wilson
Hi David,
On Thu, Jul 20, 2017, at 12:47 PM, ddw David Wilson wrote:
Mailman 3 Status Update Please
I have been following the progress and difficulties with great interest as many struggle to implement the new version of the great Mailman 3
list-server software. Last year about this time I posed some questions regarding the current development status and recognize that many additional problems seem to get in the way of actually working on long term big picture issues.From my perspective of being an senior (ie overaged) person desirous of implementing fully-featured, capable, scalable, reliable, secure applications such as Mailman, I am convinced that I am thoroughly over my head.
From my limited perspective, it would seem to me that a 'nice to have' system would incorporate the following.
- An open container standard configurable system install, compatible with different web & mail servers, DB utilities, etc;
This already exists as of today, there are some open bugs and issues but nothing that would prevent you from using it right now. You can follow the documentation for the container images here1 and look at the issues on the corresponding Github project.
It should be compatible with major web & mail server and databases.
- Python 3 version Django utilities for archiving and mail list name management;
This is in the Roadmap and on high priority now that we have Mailman 3.1 out.
- A design/plan from an inside expert perspective stating costs, time and resources to implement the plan, incorporating crowdfunding to pay for project.
- As a backup incentive to the crowdfunding plan might be the offer of an expert to set up a system for each significant amount donated, say US$1,000, such as might be accomplished by an SSL session on the target computer/server after the user has specified all necessary detail as required in a template.
AFAIK the Core developers aren't providing any services for Mailman 3 as of now.
I am sure that the program author and other major contributors could greatly enhance this list.
In summary, I guess what I am requesting is an overview forecast by half year period and out for the next two years?
Please share the big picture?
There are some TODO items that we have talked about but they aren't set as hard goals with deadlines.
Most important of it is adding more features to Postorius (the Web Frontend) so as to accomplish everything one would want to interact with Mailman Core. One missing feature of the top of my head is the customizing the template.
Fixing the upgrade path from Mailman 2 is another important issue.
Other changes have more to do with the quality of code and probably more robust testing of the different components.
Another important thing would be to have more user documentation for different management related tasks, like a List Administrator Manual.
Apart from this, I can't think anything more as of now. Others might step in if they feel there is something missing in the list.
-- thanks, Abhilash Raj
I have maybe just a little bit to add to what Abhilash has said.
On Jul 20, 2017, at 14:47, ddw David Wilson <ddw.256@gmail.com> wrote:
- A design/plan from an inside expert perspective stating costs, time and resources to implement the plan, incorporating crowdfunding to pay for project.
- As a backup incentive to the crowdfunding plan might be the offer of an expert to set up a system for each significant amount donated, say US$1,000, such as might be accomplished by an SSL session on the target computer/server after the user has specified all necessary detail as required in a template.
We do actually have a Mailman donation fund, set up through the Free Software Foundation. After an initial seed, and some occasional funds from GSoC, we have a little bit of money that we’ve used to help GSoC students attend Pycon, where most of the in-person hacking happens.
https://my.fsf.org/civicrm/contribute/transact?reset=1&id=22
I encourage everyone who loves Mailman (of any version) to donate! You help us and the FSF.
In summary, I guess what I am requesting is an overview forecast by half year period and out for the next two years?
Please share the big picture?
I would encourage everyone who wants a turnkey Mailman 3 system to help Abhilash and others to test and contribute to his docker image project. That’s really the officially sanctioned best way to deploy Mailman for those who don’t want to “roll their own”.
There are also folks in Debian working on getting all the packaging set up for Mailman 3. Right now there are some dependencies that are working their way through the process first. Hopefully, we’ll also start to see Mailman 3 become available on other Linux distros too.
Each of the subprojects has their own rough set of plans, and the best way to help is through the various Gitlab projects under this umbrella:
For Core, I’ve started to milestone some issues and merge proposals to 3.2.
One big project that I’d love to see gain more traction is Lemme, our authenticating REST proxy. We just haven’t had much collective resources to spend working on it.
https://gitlab.com/mailman/lemme
Cheers, -Barry
On 21-Jul-17 22:58, Barry Warsaw wrote:
One big project that I’d love to see gain more traction is Lemme, our authenticating REST proxy. We just haven’t had much collective resources to spend working on it.
https://gitlab.com/mailman/lemme
Cheers, -Barry
It's not obvious why this is a 'big project', or why it needs to be a separate proxy. Modularity can go too far.
The architectural problem with a separate proxy that provides separation of privilege is that the proxy has to be kept in sync with the core - add an object or property to the core, and the proxy has to be updated. You'll end up mirroring the decoding and structure of the core in the proxy in order to figure out what the access rights are. Or the proxy does just authentication and passes on a token used for authorization - in which case the proxy isn't adding much value. You already have a database of users & passwords - adding another database for the proxy is not going to simplify the admin's life.
In addition, as the recent issues indicate, the REST API already has scaling problems - adding a proxy in the path will exacerbate those.
The REST API supports Basic authentication - which is adequate for identifying users and delivering passwords, if over an encrypted link (e.g. TLS, which is easy to set up with standard libraries). Personally, I prefer client certificates as an identifier - which again, your TLS library will validate. They can be mapped to a username easily
- just add the certificate's issuer and subject to the user data. TLS will handle verification & revocation (CRL and/or OCSP with your "trusted authorities" bundle or path). Webservers handle this, and can be used as a model. (E.g. apache's mod_ssl).
What's missing is a STARTTLS-like mechanism in the REST API, a privileges field in the user and/or list databases, and suitable checks in the API actions. Even that can be simplified by requiring TLS for all connections, even local ones. I prefer that approach, as it reduces bugs and corner cases. For backward compatibility, a global configuration setting for TLS on or off can be added.
Alternatively, you can use a webserver as a reverse proxy & let it handle the authentication. You already do this with DJango. The drawback is that the path from the proxy to the mailman REST server needs to be protected or trusted - a local socket on the same machine as the webserver is one approach. But that limits configurations. Or you can use TLS from the webserver to the API - but setting that up (and documenting it for users) is more work - and at that point, you've done 95% of the work for a direct TLS connection. (The advantage is where the webserver provides single signon authentication.) So I'd avoid this path & implement TLS in the REST server.
Either way, the major work is (or should be) authorization tests in the core, with a touch of mapping an authenticated user to her(his) capabilities (for a list) in the database. Not a s"big" separate project.
Why have you come to a different conclusion?
On Jul 22, 2017, at 06:58, tlhackque via Mailman-users <mailman-users@mailman3.org> wrote:
On 21-Jul-17 22:58, Barry Warsaw wrote:
One big project that I’d love to see gain more traction is Lemme, our authenticating REST proxy. We just haven’t had much collective resources to spend working on it.
The architectural problem with a separate proxy that provides separation of privilege is that the proxy has to be kept in sync with the core -
This can be true, as is the case for example in mailmanclient. I think that’s different than the lemme case though because mmclient is a language binding to a REST API, so it needs to know the details of the API in order to provide a high-fidelity binding. This could potentially be mitigated if we had a static representation of the API, and there have been discussions of that elsewhere, so let’s not dive too deeply into that discussion here.
Where I think lemme is different is that it would be a more pure HTTP command forwarding proxy with a bit of preprocessing. It wouldn’t need to know the details of the REST API and wouldn’t need to be explicitly kept in sync.
The general approach we hashed out a few Pycons ago is outlined here: https://gitlab.com/mailman/lemme/blob/master/OUTLINE.rst
In addition, as the recent issues indicate, the REST API already has scaling problems - adding a proxy in the path will exacerbate those.
Yes, the proxy will not be very fast, but it’s also targeting a very different audience.
The current REST API, what I call the “administrative” API needs to be fast (and I’m confident we can make improvements to the existing implementation) because it’s there to serve Postorius, HyperKitty, and any other custom, tightly integrated front-ends. It provides full access to the API with no authorization because it trusts those front-ends implicitly, and those front-ends need the ability to perform any necessary action.
Lemme must fundamentally *not* trust its clients. Authentication is only the first part of its paranoia; it must also authorize the authenticated user and determine whether they have permissions to perform the requested action. It’s the authorization part that makes this a “big project” to me.
Either way, the major work is (or should be) authorization tests in the core, with a touch of mapping an authenticated user to her(his) capabilities (for a list) in the database. Not a s"big" separate project.
I disagree about the bigness, but I can be convinced with code. :)
The core has no authorization or permission information built-in. It does have a user database, but I’ve always considered that to be tightly focused on the data needed to determine posting permissions, not access to the REST API. We also don’t have a model for how users map to API permissions. We’ve always recommended that the admin API be exposed only on trusted IPs (highly recommending localhost), but a proxy has to be exposed on the public internet for it to be useful. That, and because the admin API is default-open and the proxy must be default-closed, we want to segregate any possible vulnerabilities into an optional component. Not every site will want to allow their list operations to be scriptable.
To me, this all points to a separate proxy, developed as an official subproject. There we can experiment with permission models to find what works best and is the most understandable. We can allow others to experiment as well, and sites can opt into or out of running the proxy, or even the official proxy. Any vulnerabilities in the proxy can be fixed quickly, and rolled out on a different release schedule than Core.
Now, we’re careful, I think the bulk of lemme could be written with an eye toward some future integration with Core, although I would strongly recommend running it as a separate HTTP server than the admin API. That way, we can experiment with and develop lemme at its own pace, and once it stabilizes, we could consider bringing it in as a Core feature.
Cheers, -Barry
Adding a slightly different perspective to Abhilash's and Barry's.
ddw David Wilson writes:
From my perspective of being an senior (ie overaged) person desirous of implementing fully-featured, capable, scalable, reliable, secure
I'm not sure if "secure" is a real requirement, a nice-to-have, or intended as a compliment to Mailman, but "secure" is not something we can claim. Mail is inherently a security problem, because there is no universal way to authenticate senders. It's possible under certain conditions to configure a system to be secure in the sense of allowing only authentic senders, but even when possible it is not easy. Ask John Podesta (for a well-known example) or Matt Tait (aka Twitter's @pwnallthethings, a security expert who has publically admitted that he's come "this close" to clicking on spear-phishing attachments).
Mailman itself is a complex application with a very large attack surface. If containerized, it will be rather difficult to exploit the rest of the system, but Mailman itself contains a lot of private data (email addresses and associated display names, which are frequently real names), and in enterprise situations may have access to enterprise databases "containing the whole store".
applications such as Mailman, I am convinced that I am thoroughly over my head.
That's what we're here for! Of course we're volunteers, so our availability is less than 99% reliable and time is limited. I don't know of any consulting services providing Mailman 3 consulting yet, but given the activity of this list I believe we'll see them soon. If you have funding to offer and want to kickstart the process, you could go to the vendors page on our website and talk to some of them.
From my limited perspective, it would seem to me that a 'nice to have' system would incorporate the following.
- An open container standard configurable system install, compatible with different web & mail servers, DB utilities, etc;
We have a container system which is configurable. However, IMHO "standard" and "configurable/compatible" are incompatible goals. What we can reasonably provide is a full system-in-a-box for people who only want to configure Mailman itself (ie, add lists and users, and configure their settings in the web interfaces). For people with specific desiderata such as a particular mail transport agent or web server different from those provided by our "Mailman-3-in-a-box", they're going to need somewhat advanced sysadmin skills for the forseeable future because the configuration requirements of web services and mail services must be coordinated, and have subtle, implementation-dependent interactions.
We hope that the vendors referred to above (including mail hosts, mail consultants, and system management framework vendors like cPanel) will soon start providing more "Mailman 3 in a box" services (both software distributions and as VMs in cloud services), as well as consulting services for those with specific needs not served by existing containerized versions.
- Python 3 version Django utilities for archiving and mail list name management;
In progress.
- A design/plan from an inside expert perspective stating costs, time and resources to implement the plan, incorporating crowdfunding to pay for project.
We don't have spare cycles at the moment. None of us actively consult for money on Mailman, which means neither direct engineering capability nor management capability are available. I'm considering this, but since I'm currently a life-long academic with a full-time job for the next few years, I'm not in a big hurry, and would need to acquire skills/partners for the aspects I've no experience in. :-) (I guess that's :-( from your perspective.) With a two-year timeline, if I decide to do it, likely I'd actually open for business in about 1.5-2 years (based on seasonal cycles in free time). I don't know offhand of anyone else even thinking about it.
This could change at any time with entry of new developers, of course, but it's only fair to warn you that this is the current status.
- As a backup incentive to the crowdfunding plan might be the offer of an expert to set up a system for each significant amount donated, say US$1,000, such as might be accomplished by an SSL session on the target computer/server after the user has specified all necessary detail as required in a template.
We do have ways to donate via our relationship with the FSF, but the same issues described above (no reliable gobs of available time for project oversight or actual development) apply.
I am sure that the program author and other major contributors could greatly enhance this list.
In summary, I guess what I am requesting is an overview forecast by half year period and out for the next two years?
For 6 and 12 months out, Abhilash's summary list, and Barry's, all are goals I support. We should also have a plug-in implementating the ARC protocol which improves authentication capabilities by allowing a list's subscribers (more precisely, their mail hosts) to extend transitive trust to senders that the list has authenticated. Although providing such authentication services *in Mailman* is suboptimal and sometimes impossible, we can provide containerized Mailman with that service.
Steve
-- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
Thanks to the three quality responses to my request for a status update. All respondents clearly know about which they speak, which is certainly more than I can say for myself.
From the excellent quality and quantity of the recent messages going through this mailman-users group, it's easy to be excited about the wonderful, thriving development & implementation community. Wow! [Technical term]
As a casual, very interested observer, it's nice to see this up-tick (IMO) in activity, suggesting great advances are imminent:-) in this software, originally started over a decade ago.
In this environment of ever increasing rate of change with a corresponding increase in complexities, number of dependencies, etc, I continue be impressed how Barry et al can stay the course, moving slowly and steadily (hopefully) towards the objective of 'having everything in the box' that 'just works'. That may be a somewhat unrealistic goal for such a multi-faced program, configurable for the myriad of user requirements.
My own goal is to implement and train others (once I learn how to do it myself) to use Mailman3 for various cottage and/or condo/HOA associations, each with their own lists of members, an executive board, (sub)committees, special interest groups, and so on with many common communication/data processing requirements.
Your point regarding security of messaging is appreciated but until the average email incorporates something like PGP, then messages themselves will be insecure. Since I am not contemplating a banking system, I am not unduly concerned direct about fiscal consequences such as revealing credit card numbers and/or social insurance numbers..
Regarding financial contributions, until I have a regular revenue
stream, anticipated to be priced out at approximately $25 per month per
organization, given that we will be operating as a not-for-profit
entity, I only spend/invest my own money only to specifically move
towards that goal. I was referring to money in exchange for specific,
targeted, measurable goals designed to get us going but can see how that
would not be feasible with the vast scope of this immense project.
Once up and running with a critical mass of users, we anticipate spending/returning about one third of revenue to various OSS software projects, of which Mailman3 is expected to figure prominently.
Personally, when it comes to contributing, I like to have my funds tied or directed to specific, measurable, delivered goals such as frequently seen with crowdfunding applications, rather than simply contributing to an open pool of cash, not earmarked for something specific. Just me, perhaps, but tying the dollars to results makes me feel good, even if it's awkward for those who simply want to delve into the code to fix an irritating bug or work on the next inter-application interface, etc.
As an example of identifying specific, identifiable goals to motivate donations, I helped start a charity poker run fund raiser for our local hospital, with all funds allocated to purchase specific items, including a digital mammography unit and now stretcher beds for the emergency room, with our four year total contribution approaching CDN$50,000. And all this money from a rural community numbering under 4,000 people.
I gather that with the high quality of the talent pool contributing to this user group that volunteer programmer time is more important than money, recognizing that having more money is never a bad thing.
I started this message, and plan to end it as well, with many thanks to the various respondents and expressed how impressed I am with the quality and knowledge apparent from both those asking and answering questions. The progress leads me to believe that in the foreseeable future, that this suite of programs will workable for me, with a little/lot of help from my friends:-).
Keep up the great work.
Your cheerleader,
Dave Wilson
On 17-07-23 09:53 PM, Stephen J. Turnbull wrote:
Adding a slightly different perspective to Abhilash's and Barry's.
ddw David Wilson writes:
From my perspective of being an senior (ie overaged) person desirous of implementing fully-featured, capable, scalable, reliable, secure
I'm not sure if "secure" is a real requirement, a nice-to-have, or intended as a compliment to Mailman, but "secure" is not something we can claim. Mail is inherently a security problem, because there is no universal way to authenticate senders. It's possible under certain conditions to configure a system to be secure in the sense of allowing only authentic senders, but even when possible it is not easy. Ask John Podesta (for a well-known example) or Matt Tait (aka Twitter's @pwnallthethings, a security expert who has publically admitted that he's come "this close" to clicking on spear-phishing attachments).
Mailman itself is a complex application with a very large attack surface. If containerized, it will be rather difficult to exploit the rest of the system, but Mailman itself contains a lot of private data (email addresses and associated display names, which are frequently real names), and in enterprise situations may have access to enterprise databases "containing the whole store".
applications such as Mailman, I am convinced that I am thoroughly over my head.
That's what we're here for! Of course we're volunteers, so our availability is less than 99% reliable and time is limited. I don't know of any consulting services providing Mailman 3 consulting yet, but given the activity of this list I believe we'll see them soon. If you have funding to offer and want to kickstart the process, you could go to the vendors page on our website and talk to some of them.
From my limited perspective, it would seem to me that a 'nice to have' system would incorporate the following.
- An open container standard configurable system install, compatible with different web & mail servers, DB utilities, etc;
We have a container system which is configurable. However, IMHO "standard" and "configurable/compatible" are incompatible goals. What we can reasonably provide is a full system-in-a-box for people who only want to configure Mailman itself (ie, add lists and users, and configure their settings in the web interfaces). For people with specific desiderata such as a particular mail transport agent or web server different from those provided by our "Mailman-3-in-a-box", they're going to need somewhat advanced sysadmin skills for the forseeable future because the configuration requirements of web services and mail services must be coordinated, and have subtle, implementation-dependent interactions.
We hope that the vendors referred to above (including mail hosts, mail consultants, and system management framework vendors like cPanel) will soon start providing more "Mailman 3 in a box" services (both software distributions and as VMs in cloud services), as well as consulting services for those with specific needs not served by existing containerized versions.
- Python 3 version Django utilities for archiving and mail list name management;
In progress.
- A design/plan from an inside expert perspective stating costs, time and resources to implement the plan, incorporating crowdfunding to pay for project.
We don't have spare cycles at the moment. None of us actively consult for money on Mailman, which means neither direct engineering capability nor management capability are available. I'm considering this, but since I'm currently a life-long academic with a full-time job for the next few years, I'm not in a big hurry, and would need to acquire skills/partners for the aspects I've no experience in. :-) (I guess that's :-( from your perspective.) With a two-year timeline, if I decide to do it, likely I'd actually open for business in about 1.5-2 years (based on seasonal cycles in free time). I don't know offhand of anyone else even thinking about it.
This could change at any time with entry of new developers, of course, but it's only fair to warn you that this is the current status.
- As a backup incentive to the crowdfunding plan might be the offer of an expert to set up a system for each significant amount donated, say US$1,000, such as might be accomplished by an SSL session on the target computer/server after the user has specified all necessary detail as required in a template.
We do have ways to donate via our relationship with the FSF, but the same issues described above (no reliable gobs of available time for project oversight or actual development) apply.
I am sure that the program author and other major contributors could greatly enhance this list.
In summary, I guess what I am requesting is an overview forecast by half year period and out for the next two years?
For 6 and 12 months out, Abhilash's summary list, and Barry's, all are goals I support. We should also have a plug-in implementating the ARC protocol which improves authentication capabilities by allowing a list's subscribers (more precisely, their mail hosts) to extend transitive trust to senders that the list has authenticated. Although providing such authentication services *in Mailman* is suboptimal and sometimes impossible, we can provide containerized Mailman with that service.
Steve
Thanks for the kind words, David. I just want to mention one thing.
On Jul 24, 2017, at 16:42, ddw David Wilson <ddw.256@gmail.com> wrote:
Personally, when it comes to contributing, I like to have my funds tied or directed to specific, measurable, delivered goals such as frequently seen with crowdfunding applications, rather than simply contributing to an open pool of cash, not earmarked for something specific. Just me, perhaps, but tying the dollars to results makes me feel good, even if it's awkward for those who simply want to delve into the code to fix an irritating bug or work on the next inter-application interface, etc.
Very understandable, and we’ve done this in the past. Our initial donation seed money came from Control.com to implement the topics feature of Mailman 2.
It’s not unfeasible to do something similar today, although as always, such funding focused development work has to serve the needs of our users and overall project goals. We wouldn’t take money to implement something completely antithetical to our vision or principles.
An example of something worthy to fund would be the dynamic lists/nosey lists feature that Systers pioneered, porting that to Mailman 3 proper. But funded development work is difficult because it requires a motivated donor, developer resources (i.e. time), and a project need.
One important thing that our pool of donations has supported in the past has been bringing GSoC students to Pycon to sprint with the core team. We need only point to Abhilash Raj, who started out as a GSoC student and is now a core developer working on amazing things. Talk about the multiplier effect! :)
For many of us (well, I’ll speak for myself), Mailman has been a labor of love. It’s hard to believe it’s been something near 20 years, and somehow it’s still fun!
-Barry
participants (6)
-
Abhilash Raj -
Barry Warsaw -
Barry Warsaw -
ddw David Wilson -
Stephen J. Turnbull -
tlhackque