Rolling releases for Container Images and Funding Campaign for Mailman
Hi Everyone,
As you all know I have been working on container images for Mailman 3. We now have a new "rolling" tag for both mailman-core and mailman-web images. These images have latest source installed for every Mailman component. You can find more information about them on the website 3.
New images are available on quay.io and, moving forward, the rest of the image builds will also be moved to Quay4.
These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :)
As most of you already know, Mailman 3 is the new and improved version with extra features, better security and much better architecture. We released Mailman Suite 3.0 in April 2015 and have come a long ways since then. Mailman Suite 3.1, release May 2016, was aimed to provide feature-parity with Mailman 2.x series and we think we _almost_ hit that goal.
Apart from no monthly password reminders, Mailman 3 has a much better Administrator/User interface, REST API for scripting, a really awesome archiver, support for multiple domains, support for external plugins, support for SSO/social login and so much more!
I love working on Mailman and would enjoy being able to do so full time for next 6-8 weeks. Mailman 3 is not very far from becoming the default version everyone would use, but it still needs some work to get there. I need help from you, the users of Mailman, to get us there. If you or your organization would like to move to (or, already moved to) Mailman 3, I urge you to donate1 to us.
There are options to donate using Credit Card, Paypal, Bitcoin, Wire Transfer (of any currency), Check and money order.
If this campaign succeeds, here is a road map of what I intend to get done:
- Move Django apps(UI/Archiver) to Python 3 (or bilingual)
- Fork
mailman importcommand to provide an upgrade path to Mailman 3.x from Mailman 2.x - Fix MySQL compatibility in Core
- Changes in Postorius:
- Add support for missing options that are already exposed in Core’s
API
- e.g. Support for setting templates
- Find the commonly used options that are not exposed in Core, add them to Core's API and add to Postorius
- Add Admin Dashboard project from GSoC 2014 (maybe?)
- Add support for missing options that are already exposed in Core’s
API
- Add better testing of container images and provide deployment instructions for Kubernetes & Docker Swarm
- Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services.
- Administrator/User documentation for Postorius & Mailman
- (optional) Fork mmcli project from Rajeev, fix if there is anything missing and add it as an additional command line tool to work with Mailman Core. Maybe pull it under Mailman umbrella.
Except for these, if there is something more important that is preventing the adoption of Mailman 3 from your end, we can discuss them. I'd like to mention that I have been working on Mailman 3 for quite some time now and I intend to implement every single item on the list. You donations would help it get done much sooner, hopefully in time for 3.2 release schedule (at PyCon US 2018).
You can follow the progress of this campaign here2.
-- Abhilash Raj maxking@asynchronous.in
On 11/08/2017 12:03 AM, Abhilash Raj wrote:
New images are available on quay.io and, moving forward, the rest of the image builds will also be moved to Quay[4][5]. Since docker deployment is currently the recommended install, with manual installs using the master branches for "experts" I wonder why you are using Quay. It looks like it's a non-free system. Why use Quay but refuse to use a non-free translation system like transifex? These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :) This contradicts what you said in my merge request for Postorius https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756
You either use released versions, or make sure that the master branches are always compatible. You can't have both with the current development model.
If this campaign succeeds, here is a road map of what I intend to get done: Sounds great, what is the limit where the campaign succeeds? What will happen to the funds if it doesn't?
- Add Admin Dashboard project from GSoC 2014 (maybe?) Didn't find anything using google. What is that project about?
- Add better testing of container images and provide deployment instructions for Kubernetes & Docker Swarm To be honest, I don't think we should all in on containers. Yes they are nice, but I guess most people would prefer distro packages. Even if people want containers, I for instance would prefer plain systemd-nspawn
- Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services. The current bottleneck is the rest api from core. IMO bulk actions, filtering and sorting should be added there before trying to have multiple postorius/hyperkitty instances serving the same core...
On Nov 8, 2017, at 04:38, Simon Hanna simon@hannaweb.eu wrote:
On 11/08/2017 12:03 AM, Abhilash Raj wrote:
New images are available on quay.io and, moving forward, the rest of the image builds will also be moved to Quay[4][5]. Since docker deployment is currently the recommended install, with manual installs using the master branches for "experts" I wonder why you are using Quay. It looks like it's a non-free system. Why use Quay but refuse to use a non-free translation system like transifex?
Is there a free (software) equivalent to Quay? Caveat: I don’t know anything about Quay, so I don’t know what services and advantages it provides.
If this campaign succeeds, here is a road map of what I intend to get
done: Sounds great, what is the limit where the campaign succeeds? What will happen to the funds if it doesn’t?
The funds all go to the GNU Mailman project’s FSF donation fund. A portion of every donation goes to the FSF, and the Mailman Steering Committee ultimately decides what to do with what’s left in our account. In the past we’ve used it sponsor GSoC students coming to Pycon sprints.
To be honest, I don't think we should all in on containers. Yes they are nice, but I guess most people would prefer distro packages. Even if people want containers, I for instance would prefer plain systemd-nspawn
There are folks working on Debian packages. I’m not aware of similar efforts for other distro ecosystems. But I don’t think it’s all or nothing, and it does make sense for us (the GNU Mailman project) to support containers to help with adoption, experimentation, and deployment, while working with distro volunteers to package the code up for those platforms.
Cheers, -Barry
On Wed, Nov 8, 2017, at 04:38 AM, Simon Hanna wrote:
On 11/08/2017 12:03 AM, Abhilash Raj wrote:
New images are available on quay.io and, moving forward, the rest of the image builds will also be moved to Quay[4][5]. Since docker deployment is currently the recommended install, with manual installs using the master branches for "experts" I wonder why you are using Quay.
For those who don't know about it, Quay is a container registry offering from CoreOS. The main reason why I chose to move to Quay, from DockerHub that we are currently using, is security.
Some parts of both Dockerhub and Quay are open source and some aren't. There isn't a very clear boundary/transparency in what technologies they use. Both are free to use for open source projects. However, Quay provides an improved security model with RBAC to push images from build server, without having to put my account password on the build server. It also includes a (open source) security scanner for images, which looks for vulnerabilities in packages included in the image so that I can re-build them if one is found.
There are open source registries available, most popular of which I know is provided by Gitlab. However, I feel Quay is better for my current workflow, which involves lots of bash scripts for building and testing images on public CI system for rolling releases.
Given more free cycles, I do intend to have a better build pipeline and possibly use Gitlab's own registry to distribute images.
It looks like it's a non-free system. Why use Quay but refuse to use a non-free translation system like transifex?
These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :) This contradicts what you said in my merge request for Postorius https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756
You either use released versions, or make sure that the master branches are always compatible. You can't have both with the current development model.
Not exactly, I still believe all our git-heads should be compatible with released (i.e. stable) versions of dependencies, regardless of the fact that we maintain those dependencies.
Also, there is at least one test per-project which tests with git-heads of dependencies we maintain, so that we know if we make any incompatible changes.
I am not sure what in the current development model prevents us to do both?
The story for container images is different than testing with git-heads. We need better integration testing for the entire suite to be stable in conjunction, rather than independently. In fact, containers might be the perfect way to actually do the integration testing.
If this campaign succeeds, here is a road map of what I intend to get done: Sounds great, what is the limit where the campaign succeeds? What will happen to the funds if it doesn't?
- Add Admin Dashboard project from GSoC 2014 (maybe?) Didn't find anything using google. What is that project about?
https://www.google-melange.com/archive/gsoc/2015/orgs/gnu_mailman/projects/b...
- Add better testing of container images and provide deployment instructions for Kubernetes & Docker Swarm To be honest, I don't think we should all in on containers. Yes they are nice, but I guess most people would prefer distro packages. Even if people want containers, I for instance would prefer plain systemd-nspawn
Like Barry mentioned, it doesn't have to be all-or-nothing. There is work being done on distro packaging for Debian and I am ready to help others trying to do the same for other distros!
Systemd-nspawn isn't really a production grade tool to use containers. I am not sure about how much it adheres to the OCI specs, but if it does, it should not matter which deployment tool you are using with images.
I wouldn't mind adding configurations for systemd-nspawn in the documentation, contributions are most welcome.
- Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services. The current bottleneck is the rest api from core. IMO bulk actions, filtering and sorting should be added there before trying to have multiple postorius/hyperkitty instances serving the same core...
I didn't say they will be served from the same core ;-)
Yes, the Core's API can be further improved to be more efficient. But that doesn't prevent us from scaling. I understand there are issues around multiple instances of containers, but I have ideas to get around most of them.
-- Abhilash Raj maxking@asynchronous.in
These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :) This contradicts what you said in my merge request for Postorius https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756
You either use released versions, or make sure that the master branches are always compatible. You can't have both with the current development model. Not exactly, I still believe all our git-heads should be compatible with released (i.e. stable) versions of dependencies, regardless of the fact that we maintain those dependencies.
Also, there is at least one test per-project which tests with git-heads of dependencies we maintain, so that we know if we make any incompatible changes.
I am not sure what in the current development model prevents us to do both?
The story for container images is different than testing with git-heads. We need better integration testing for the entire suite to be stable in conjunction, rather than independently. In fact, containers might be the perfect way to actually do the integration testing. Scenario: Core changes something about rest that is backwards incompatible. The change is commited to master. Since your containers use the master branches, all future builds will fail until mailmanclient and postorius/hyperkitty. If you want to always have Postorius (and others) compatible with the latest release of master, development for Postorius will be a Pain, because you can't use the master branch of mailman for testing and quick changes. Also your containers and all integration tests that can be done with them will always fail.
Until you fix Postorius and the other clients... So why not just have the master branches always compatible and make life easier for developers. I don't see a good reason why you want to do it the other way round...
Any bug in core that affects Postorius will also just remain there until you release a new version of core.
If you really really want to have it your way, there have to be more frequent releases.
- Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services. The current bottleneck is the rest api from core. IMO bulk actions, filtering and sorting should be added there before trying to have multiple postorius/hyperkitty instances serving the same core... I didn't say they will be served from the same core ;-)
Yes, the Core's API can be further improved to be more efficient. But that doesn't prevent us from scaling. I understand there are issues around multiple instances of containers, but I have ideas to get around most of them. I just don't want that people that have issues with big installations get the answer: We know of that issue, please deploy Mailman on X containers/machines as a workaround
On Sun, Nov 12, 2017, at 08:06 AM, Simon Hanna wrote:
These images are built using git-heads *only* if they are passing our test suite and are re-generated weekly. You should be aware that while all these components are tested with their individual test suites, their combination might sometimes not be stable. This will get you updates/bug-fixes much faster :) This contradicts what you said in my merge request for Postorius https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756
You either use released versions, or make sure that the master branches are always compatible. You can't have both with the current development model. Not exactly, I still believe all our git-heads should be compatible with released (i.e. stable) versions of dependencies, regardless of the fact that we maintain those dependencies.
Also, there is at least one test per-project which tests with git-heads of dependencies we maintain, so that we know if we make any incompatible changes.
I am not sure what in the current development model prevents us to do both?
The story for container images is different than testing with git-heads. We need better integration testing for the entire suite to be stable in conjunction, rather than independently. In fact, containers might be the perfect way to actually do the integration testing. Scenario: Core changes something about rest that is backwards incompatible. The change is commited to master. Since your containers use the master branches, all future builds will fail until mailmanclient and postorius/hyperkitty.
Yes, that is a totally possible scenario.
But, given that kind of testing we currently do, this scenario can happen not only to git-heads but also releases. That is what I am trying to build using container images.
The fact that I *announced* and *released* these rolling images for public use, is only because people asked for it before. These images come with a fair amount of warning that, right now, they can break!
If you want to always have Postorius (and others) compatible with the latest release of master, development for Postorius will be a Pain, because you can't use the master branch of mailman for testing and quick changes. Also your containers and all integration tests that can be done with them will always fail.
It would most definitely be a pain if we were to break REST API frequently, but that is quite rare. Given that in future API will be consumed by not just us, it is important to keep this assumption true.
Another wishful project I have is to be able to "map" the REST API and monitor for changes; it'd also help us with lemme (the authenticating/authorization proxy for Core).
And yes, container images will fail, and that's the point! I want them to fail so that releases don't :)
Rolling images are meant to bring a faster feedback-loop from users who are OK with rare occasional breaking and want to help test the software, something like public-beta releases.
Until you fix Postorius and the other clients... So why not just have the master branches always compatible and make life easier for developers.
Incompatible changes are released only with major version upgrades, so it is not going to happen with everyone, unless they choose to use the git-heads and get themselves into it.
I don't see a good reason why you want to do it the other way round...
Any bug in core that affects Postorius will also just remain there until you release a new version of core.
If you really really want to have it your way, there have to be more frequent releases.
Yes, we definitely don't want to say "use new beta-quality container images or stone age stable release". Going forward, we will have more frequent releases too :)
- Improve the container images to work with new micro-services architecture, to achieve scaling and redundancy in services. The current bottleneck is the rest api from core. IMO bulk actions, filtering and sorting should be added there before trying to have multiple postorius/hyperkitty instances serving the same core... I didn't say they will be served from the same core ;-)
Yes, the Core's API can be further improved to be more efficient. But that doesn't prevent us from scaling. I understand there are issues around multiple instances of containers, but I have ideas to get around most of them. I just don't want that people that have issues with big installations get the answer: We know of that issue, please deploy Mailman on X containers/machines as a workaround
Containers are just an easy way to scale, given that we already have them, it'd be fun to see if I can make multiple instances run in parallel. Given that _most_ parts store their data in a data store (database, redis etc), it shouldn't be too difficult. There will be quirks with logging, queues in Core and hence held message views in Postorius, but apart from that, I think everything else can be served using any random instance of Core/Postorius/Hyperkitty.
We can most definitely run multiple instances of Hyperkitty, which I feel will be more used than any other part of UI. If we partition the data smartly, we could do the same with Core.
Again, these are just theories and needs much more testing and some changes in components themselves! This would most definitely increase the availability and auto-scaling.
Making Core's API more efficient would definitely be a good thing and we do intend to do that in future. But, this would still be relevant if we had an efficient REST API, it's not a band-aid to the actual problem :)
-- Abhilash Raj maxking@asynchronous.in
On Nov 12, 2017, at 11:06, Simon Hanna simon@hannaweb.eu wrote:
Scenario: Core changes something about rest that is backwards incompatible. The change is commited to master.
The REST API is versioned, so it would be a bug to introduce a backward incompatibility in the same API version. That’s why for example a new API version was introduced to handle the UUID data type change. There was no way to make that backward compatible, so we had to bump the API version, but we didn’t remove the old API version so clients written against that still work. Adding a new API generally doesn’t require bumping the API version.
Cheers, -Barry
participants (3)
-
Abhilash Raj -
Barry Warsaw -
Simon Hanna