Upgrade-path from Ubuntu18LTS to recent mailman release
Dear Users,
we are very happy mailman3-users but suffer from very old packages in Ubuntu 18LTS.
Unfortunately due to some python-restrictions¹, mailman3 is broken in Ubuntu20LTS and gone² in Ubuntu 22LTS, so we do not have any working upgrade-path :/
What do you recommend?
Any help is greatly appreciated.
thanks in advance.
Stefan
¹ https://bugs.launchpad.net/ubuntu/+source/mailman3/+bug/1877701 ² https://packages.ubuntu.com/search?keywords=mailman3-full
I followed this guide to install mailman3 in a virtualenv on Debian:
https://docs.mailman3.org/en/latest/install/virtualenv.html[1]
This way you are independent of the release cycle of your distribution. Works fine, you just have to adjust the pathes in your config files.
[1] https://docs.mailman3.org/en/latest/install/virtualenv.html
On 5/3/22 01:32, Eggert Ehmke wrote:
Somehow the URL is broken. Just look for mailman3+virtualenv in Google.
https://docs.mailman3.org/en/latest/install/virtualenv.html works for me.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Eggert,
thanks alot. That looks promising. Can we just copy the current mailman3-database and take the configurations or is there a specific migration path we have to follow? We are currently running 3.1.1-9 (Ubuntu 18).
thanks a lot.
Stefan
Am Di., 3. Mai 2022 um 10:30 Uhr schrieb Eggert Ehmke <eggert@eehmke.de>:
I followed this guide to install mailman3 in a virtualenv on Debian:
https://docs.mailman3.org/en/latest/install/virtualenv.html[1]
This way you are independent of the release cycle of your distribution. Works fine, you just have to adjust the pathes in your config files.
[1] https://docs.mailman3.org/en/latest/install/virtualenv.html
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Hallo Stefan,
yes you can keep the database. I used the configuration as suggested in that guide and did some modifications to adapt to my older settings.
I also had the original Debian packages installed before, so my experience matches your task
pretty much.
As I am not using nginx but apache, I set up a vhost in the apache2. This setup worked for me:
https://github.com/maxking/docker-mailman/issues/525[1]
Only I did not use the uwsgi localhost:8080 address but used the sockets: ProxyPass / unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ ProxyPassReverse / unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost/ The good thing is, the installation in the virtualenv directory is completely independant of the original Ubuntu/Debian installation. You can prepare all stuff while the old instance is still running. Take care not to run both instances at the same time, this might corrupt the database.
DM me if you run into problems. Eggert
Am Mittwoch, 4. Mai 2022, 07:41:19 CEST schrieb Stefan Bauer:
Hi Eggert,
thanks alot. That looks promising. Can we just copy the current mailman3-database and take the configurations or is there a specific migration path we have to follow? We are currently running 3.1.1-9 (Ubuntu 18).
thanks a lot.
Stefan
Am Di., 3. Mai 2022 um 10:30 Uhr schrieb Eggert Ehmke <eggert@eehmke.de>:
I followed this guide to install mailman3 in a virtualenv on Debian:
https://docs.mailman3.org/en/latest/install/virtualenv.html[1]
This way you are independent of the release cycle of your distribution. Works fine, you just have to adjust the pathes in your config files.
[1] https://docs.mailman3.org/en/latest/install/virtualenv.html
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Stefan Bauer writes:
thanks alot. That looks promising. Can we just copy the current mailman3-database and take the configurations or is there a specific migration path we have to follow?
Much of the documentation on mailman.readthedocs.io discusses migrations and upgrades. You should read those for general hints.
The distro Mailman to venv Mailman case is more discussed here on the mailing list than in those documents. I don't have pointers to those threads or the time to look offhand, maybe over the next few weeks as I work on our GSoC project. If you do find any, and would post them here, I'll make sure they get into the docs.
The main thing is to backup Mailman configs (probably in /etc/mailman or /etc/mailman3) and the backend database (core, Postorius, possibly HyperKitty depending on configuration, and the django-mailman auth stuff) before trying to migrate. (For the backend database I would just dump the RDBMS unless non-Mailman content is really huge.) The nature of the virtualenv install is such that it *will not* overwrite any of the distro files at all (as long as you put it in /usr/local or /opt). However, you want to preserve the configs and database and reuse them, so those *must* be backed up.
We are currently running 3.1.1-9 (Ubuntu 18).
Ubuntu 18 is probably old enough that you need to be careful about MySQL and its derivatives. The specific problems are (1) you need to specify the right UTF-8 variant for the coding system, and (2) more recent versions of Mailman tend to make certain records overflow.
If you're using PostgreSQL these caveats *do not* apply. If you're using sqlite3 (I doubt it, but just for completeness) you should change to PostgreSQL or MySQL. If you're using something else (same caveat), I can't really say. Probably somebody out there is using Oracle or whatever, but we don't know about that. :-)
Steve
Thank you all for your time. I managed to do a fresh installation. The MM documentation is very good.
As i only use sqlite, i will give it a try to just copy the database files.
Stefan
Am Do., 5. Mai 2022 um 11:11 Uhr schrieb Stephen J. Turnbull < stephenjturnbull@gmail.com>:
Stefan Bauer writes:
thanks alot. That looks promising. Can we just copy the current mailman3-database and take the configurations or is there a specific migration path we have to follow?
Much of the documentation on mailman.readthedocs.io discusses migrations and upgrades. You should read those for general hints.
The distro Mailman to venv Mailman case is more discussed here on the mailing list than in those documents. I don't have pointers to those threads or the time to look offhand, maybe over the next few weeks as I work on our GSoC project. If you do find any, and would post them here, I'll make sure they get into the docs.
The main thing is to backup Mailman configs (probably in /etc/mailman or /etc/mailman3) and the backend database (core, Postorius, possibly HyperKitty depending on configuration, and the django-mailman auth stuff) before trying to migrate. (For the backend database I would just dump the RDBMS unless non-Mailman content is really huge.) The nature of the virtualenv install is such that it *will not* overwrite any of the distro files at all (as long as you put it in /usr/local or /opt). However, you want to preserve the configs and database and reuse them, so those *must* be backed up.
We are currently running 3.1.1-9 (Ubuntu 18).
Ubuntu 18 is probably old enough that you need to be careful about MySQL and its derivatives. The specific problems are (1) you need to specify the right UTF-8 variant for the coding system, and (2) more recent versions of Mailman tend to make certain records overflow.
If you're using PostgreSQL these caveats *do not* apply. If you're using sqlite3 (I doubt it, but just for completeness) you should change to PostgreSQL or MySQL. If you're using something else (same caveat), I can't really say. Probably somebody out there is using Oracle or whatever, but we don't know about that. :-)
Steve
Stefan Bauer writes:
As i only use sqlite, i will give it a try to just copy the database files.
OK, if sqlite works for you, I think there's no hurry (I don't know of any severe performance issues for small sites, nor security issues).
However, I still recommend considering a migration to one of the more powerful RDMBSes like PostgreSQL or MySQL for the long run, because Mailman installations can easily scale to sizes where sqlite becomes a performance bottleneck. It's not that hard to do, because Mailman (mostly via the Django and SQLAlchemy ORMs, but I think there's some "raw" SQL in Mailman) sticks to the common core for SQL, so you can dump the database as SQL commands and then restore it in another SQL database without too much fiddling. You may need some fiddling, such as the database-specific initializations such as setting the encoding for text items and ensuring that sized columns are wide enough, but mostly that will be taken care of by the ORMs.
Regards, Steve
Just for the record, I did exactly this change a while ago (because I was having issues with locking of the file, IIRC). I thought I could dump the database using sqlite's export to sql tool and then import with minimal fuss into mysql, but it didn't work straight away -- too many differences in sql types and styles. I thought to edit the dump file into mysql's style, but this resulted in other errors.
In the end I found it necessary to create a blank database using the mysql backend, then dump only the data (no table or index ops) into that. I can't quite remember if any fixup was needed but if so it was very minor.
It would be nice to have a migrate DB function within mailman (i.e. a utility that created the tables in a new db from current, then copied the data using the ORM tools, so as to avoid low-level sql issues).
Ruth
On 13/05/2022 09:25, Stephen J. Turnbull wrote:
Stefan Bauer writes:
As i only use sqlite, i will give it a try to just copy the database files.
OK, if sqlite works for you, I think there's no hurry (I don't know of any severe performance issues for small sites, nor security issues).
However, I still recommend considering a migration to one of the more powerful RDMBSes like PostgreSQL or MySQL for the long run, because Mailman installations can easily scale to sizes where sqlite becomes a performance bottleneck. It's not that hard to do, because Mailman (mostly via the Django and SQLAlchemy ORMs, but I think there's some "raw" SQL in Mailman) sticks to the common core for SQL, so you can dump the database as SQL commands and then restore it in another SQL database without too much fiddling. You may need some fiddling, such as the database-specific initializations such as setting the encoding for text items and ensuring that sized columns are wide enough, but mostly that will be taken care of by the ORMs.
-- Tel: 01223 414180 Blog: http://www.ivimey.org/blog LinkedIn: http://uk.linkedin.com/in/ruthivimeycook/
Ruth Ivimey-Cook writes:
In the end I found it necessary to create a blank database using the mysql backend, then dump only the data (no table or index ops) into that. I can't quite remember if any fixup was needed but if so it was very minor.
Yes, that sounds about right to me.
It would be nice to have a migrate DB function within mailman
Patches welcome. %-)
More seriously, you wouldn't happen to have a shell log of that operation? I mean, I have stranger things than that in my $HOME. ;-)
Steve
participantes (5)
-
Eggert Ehmke -
Mark Sapiro -
Ruth Ivimey-Cook -
Stefan Bauer -
Stephen J. Turnbull