OS upgrade - anything to take care of?
Dear MM3 users,
today I upgraded one of two servers from Ubuntu 18.04 to 20.04 and - besides some minor config updates - postfix & dovecot worked well afterwards.
So I am very optimistic that the upgrade of the second server from 18.04 to 20.04 will also work fine. But the second server is running MM3 (installed in venv) and I saw that on upgrade Python 3.8 is installed.
Is there anything I have to take care of? Anything to do / change in venv?
Kind regards Torge
On 5/13/21 7:25 AM, Torge Riedel wrote:
today I upgraded one of two servers from Ubuntu 18.04 to 20.04 and - besides some minor config updates - postfix & dovecot worked well afterwards.
So I am very optimistic that the upgrade of the second server from 18.04 to 20.04 will also work fine. But the second server is running MM3 (installed in venv) and I saw that on upgrade Python 3.8 is installed.
Is there anything I have to take care of? Anything to do / change in venv?
The venv will still run whatever Python it was built with. That's what virtualenvs are for - to isolate the software therein from external changes.
You shouldn't need to do anything in the venv for Mailman to continue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark,
While the venv will to a degree isolate you, my experience of the upgrade from 18 to 20 was that python3.6 was uninstalled and 3.8 installed. In this scenario, the mailman3 installation is subsequently broken because there is no python to run.
It *is* possible to reinstall python3.6 after the dist upgrade is complete, and I would presume that after doing so mailman3 would probably resume ok. In my case I decided to continue using 3.8, which meant, in essence, rebuilding my mailman3 install -- not fun given my state of knowledge.
I would therefore appreciate some guidance on performing the upgrade from 3.6 to 3.8 with the minimum of fuss.
[I am not sure, but it could be possible to install python3.8 while using Ubuntu 18, so that the dist upgrade never breaks anything. Worth exploring.]
Ruth
On 13/05/2021 19:02, Mark Sapiro wrote:
On 5/13/21 7:25 AM, Torge Riedel wrote:
today I upgraded one of two servers from Ubuntu 18.04 to 20.04 and - besides some minor config updates - postfix & dovecot worked well afterwards.
So I am very optimistic that the upgrade of the second server from 18.04 to 20.04 will also work fine. But the second server is running MM3 (installed in venv) and I saw that on upgrade Python 3.8 is installed.
Is there anything I have to take care of? Anything to do / change in venv? The venv will still run whatever Python it was built with. That's what virtualenvs are for - to isolate the software therein from external changes.
You shouldn't need to do anything in the venv for Mailman to continue.
-- Software Manager & Engineer Tel: 01223 414180 Blog: http://www.ivimey.org/blog LinkedIn: http://uk.linkedin.com/in/ruthivimeycook/
On 5/13/21 11:28 AM, Ruth Ivimey-Cook wrote:
Mark,
While the venv will to a degree isolate you, my experience of the upgrade from 18 to 20 was that python3.6 was uninstalled and 3.8 installed. In this scenario, the mailman3 installation is subsequently broken because there is no python to run.
Yes, I overlooked some aspects of virtualenv. Actually, the Python in the virtualenv is mostly symlinks to the Python it was built with, so whether it continues to work after a Python upgrade depends on what the symlinks are and how the Python upgrade installs.
It *is* possible to reinstall python3.6 after the dist upgrade is complete, and I would presume that after doing so mailman3 would probably resume ok. In my case I decided to continue using 3.8, which meant, in essence, rebuilding my mailman3 install -- not fun given my state of knowledge.
I would therefore appreciate some guidance on performing the upgrade from 3.6 to 3.8 with the minimum of fuss.
[I am not sure, but it could be possible to install python3.8 while using Ubuntu 18, so that the dist upgrade never breaks anything. Worth exploring.]
I haven't tested any of this, but I suspect if Python is for example, /usr/bin/python3.6 and /usr/lib/python3.6 the upgrade removes those and installs /usr/bin/python3.8 and /usr/lib/python3.8, that you could just create symlinks from /usr/bin/python3.6 and /usr/lib/python3.6 to /usr/bin/python3.8 and /usr/lib/python3.8 respectively and it would work.
On the other hand, if the symlinks in the venv are to things like /usr/bin/python3 and /usr/lib/python3 which used to point to 3.6 versions and after upgrade point to 3.8 versions, it should just work.
As far as installing Python3.8 on Ubuntu 18.04 goes, the machine I'm typing this on is Ubuntu 18.04 and has 6 different Python3 versions
mark@msapiro:~$ /usr/bin/python3.5 --version Python 3.5.2 mark@msapiro:~$ /usr/bin/python3.6 --version Python 3.6.9 mark@msapiro:~$ /usr/local/bin/python3.6 --version Python 3.6.12 mark@msapiro:~$ /usr/local/bin/python3.7 --version Python 3.7.9 mark@msapiro:~$ /usr/local/bin/python3.8 --version Python 3.8.7 mark@msapiro:~$ /usr/local/bin/python3.9 --version Python 3.9.1
and /usr/bin/python3 is a symlink to /usr/bin/python3.6 and /usr/local/bin/python3 is a symlink to /usr/local/bin/python3.9, so take your pick...
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 13/05/2021 20:06, Mark Sapiro wrote:
On 5/13/21 11:28 AM, Ruth Ivimey-Cook wrote: Yes, I overlooked some aspects of virtualenv. Actually, the Python in the virtualenv is mostly symlinks to the Python it was built with, so whether it continues to work after a Python upgrade depends on what the symlinks are and how the Python upgrade installs.
Correct.
I would therefore appreciate some guidance on performing the upgrade from 3.6 to 3.8 with the minimum of fuss.
[I am not sure, but it could be possible to install python3.8 while using Ubuntu 18, so that the dist upgrade never breaks anything. Worth exploring.] I haven't tested any of this, but I suspect if Python is for example, /usr/bin/python3.6 and /usr/lib/python3.6 the upgrade removes those and installs /usr/bin/python3.8 and /usr/lib/python3.8, that you could just create symlinks from /usr/bin/python3.6 and /usr/lib/python3.6 to /usr/bin/python3.8 and /usr/lib/python3.8 respectively and it would work.
It would be nice if that worked, and I tried it, but it doesn't work because the python system and site directories under ./lib are also versioned (eg ./lib/python3.6/site-packages), so you would have to additionally create symlinks for those and/or rebuild those.
On the other hand, if the symlinks in the venv are to things like /usr/bin/python3 and /usr/lib/python3 which used to point to 3.6 versions and after upgrade point to 3.8 versions, it should just work.
Sadly they aren't - they link to python3.6 or whatever... and even if not, the /lib/ naming gets you as well.
And even if that can be worked around (eg by renaming the mailman /lib/python3.6 folder to /lib/python3.8), the python .pyc files are /also/ versioned, so you then have to find & delete them all to force the new interpreter to rebuild them or python falls down in a heap.
As far as installing Python3.8 on Ubuntu 18.04 goes, the machine I'm typing this on is Ubuntu 18.04 and has 6 different Python3 versions
mark@msapiro:~$ /usr/bin/python3.5 --version Python 3.5.2 mark@msapiro:~$ /usr/bin/python3.6 --version Python 3.6.9 mark@msapiro:~$ /usr/local/bin/python3.6 --version Python 3.6.12 mark@msapiro:~$ /usr/local/bin/python3.7 --version Python 3.7.9 mark@msapiro:~$ /usr/local/bin/python3.8 --version Python 3.8.7 mark@msapiro:~$ /usr/local/bin/python3.9 --version Python 3.9.1
and /usr/bin/python3 is a symlink to /usr/bin/python3.6 and /usr/local/bin/python3 is a symlink to /usr/local/bin/python3.9, so take your pick...
Indeed... and it would be lovely if it was that simple, but my experience was that even if you got a good python executable running, the other versioned elements of a python install caused it to mess up.
What worked for me in the end was a combination of forcing a complete rebuild/reinstall with the mailman3 python3.8 install, followed by a bit of rejigging of file permissions and settings (probably because I'd messed something up, but maybe not).
Regards
Ruth
-- Software Manager & Engineer Tel: 01223 414180 Blog: http://www.ivimey.org/blog LinkedIn: http://uk.linkedin.com/in/ruthivimeycook/
On 5/13/21 1:58 PM, Ruth Ivimey-Cook wrote:
Sadly they aren't - they link to python3.6 or whatever... and even if not, the /lib/ naming gets you as well.
And even if that can be worked around (eg by renaming the mailman /lib/python3.6 folder to /lib/python3.8), the python .pyc files are /also/ versioned, so you then have to find & delete them all to force the new interpreter to rebuild them or python falls down in a heap.
Maybe I'm missing something, but it seems to me that if /usr/lib/python3.6 and /user/bin/python3.6 are gone, just creating symlinks from /usr/lib/python3.6 and /user/bin/python3.6 to the corresponding python3.8 directories would work.
The .pyc files in __pycache__ should not be an issue. They are versioned and placed in a __pycache__ directory specifically so multiple python versions can coexist. See <https://www.python.org/dev/peps/pep-3147/>. Python3.8 will just make new ones. If there is some issue with this, one can always do
find /path/to/venv -name '*.pyc' -delete
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Ruth Ivimey-Cook wrote:
[Regarding OS upgrades with Python venvs]
What worked for me in the end was a combination of forcing a complete rebuild/reinstall with the mailman3 python3.8 install, followed by a bit of rejigging of file permissions and settings (probably because I'd messed something up, but maybe not).
Personally I quite often rebuild the venv from scratch. It’s a case of stopping the services, moving the old venv out of the way, then creating the new one and running Pip to get the needed packages in. I've done this on a similar Ubuntu 18.04 --> 20.04 upgrade in the past as well. This also pulls in later packages for some of the other dependencies which wouldn't necessarily get caught on a standard upgrade.
Andrew.
Am 13.05.21 um 23:51 schrieb Andrew Hodgson:
Personally I quite often rebuild the venv from scratch. It’s a case of stopping the services, moving the old venv out of the way, then creating the new one and running Pip to get the needed packages in.
Hi,
nice idea. Looking at my own recorded installation instructions it should be really that easy.
By the way: My instructions still list django-paintstore to be installed for the web part. I remember that on this list it is mentioned to be no more required. Is this correct.
Kind regards Torge
Torge Riedel wrote:
Am 13.05.21 um 23:51 schrieb Andrew Hodgson:
Personally I quite often rebuild the venv from scratch. It’s a case of stopping the services, moving the old venv out of the way, then creating the new one and running Pip to get the needed packages in.
nice idea. Looking at my own recorded installation instructions it should be really that easy.
By the way: My instructions still list django-paintstore to be installed for the web part. I remember that on this list it is mentioned to be no more required. Is this correct.
I don't believe so.
I take the base settings file from here:
https://gitlab.com/mailman/mailman-suite
Then update my local settings in the local file as advised by the howtos.
I am actually working on using less packages in my venv and using some of the Debian packages along with using --system-site-packages to get some of the base packages in from Debian. I am eventually hoping to not have to install build dependencies onto the target system. Its also worth noting that you can run the --upgrade on the venv command which upgrades the Python version if that version was upgraded in-place, but I decided to go the route of re-creating the venv.
Andrew.
On 5/15/21 12:00 AM, Torge Riedel wrote:
By the way: My instructions still list django-paintstore to be installed for the web part. I remember that on this list it is mentioned to be no more required. Is this correct.
django-paintstore is not required. If you still have it in INSTALLED_APPS in your Django settings, you can remove it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 13.05.21 um 21:06 schrieb Mark Sapiro:
On the other hand, if the symlinks in the venv are to things like /usr/bin/python3 and /usr/lib/python3 which used to point to 3.6 versions and after upgrade point to 3.8 versions, it should just work.
Hi,
yes, that is the case on the machine I already upgraded. But I looked to all directories in venv and saw some "non-symlinked" python 3.6 stuff. So I will go the "new venv" approach Andrew mentioned. Just to be safe.
Kind regards Torge
participants (4)
-
Andrew Hodgson -
Mark Sapiro -
Ruth Ivimey-Cook -
Torge Riedel