Postorius is not working (throws "Error 500" suddenly after changing POSTORIUS_TEMPLATE_BASE_URL and changing back)
After having changed and re-changed "only" the line in /etc/mailman3/mailman-web.py (which looks to be only a "blinder" and the real problem could be a different one)
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost/mailman3/'
the web access via Postorius is not working any more and throws "Error 500"
My logfile /var/log/mailman3/web/mailman-web.log shows now
File "/usr/share/mailman3-web/manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.9/dist-packages/django/core/management/__init__.py", line 420, in execute django.setup() File "/usr/local/lib/python3.9/dist-packages/django/__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/local/lib/python3.9/dist-packages/django/apps/registry.py", line 116, in populate app_config.import_models() File "/usr/local/lib/python3.9/dist-packages/django/apps/config.py", line 269, in import_models self.models_module = import_module(models_module_name) File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1030, in _gcd_import File "<frozen importlib._bootstrap>", line 1007, in _find_and_load File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 680, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 790, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/lib/python3/dist-packages/django_q/models.py", line 12, in <module> from django_q.signing import SignedPackage File "/usr/lib/python3/dist-packages/django_q/signing.py", line 7, in <module> from django_q import core_signing as signing File "/usr/lib/python3/dist-packages/django_q/core_signing.py", line 9, in <module> from django.utils.encoding import force_bytes, force_str, force_text ImportError: cannot import name 'force_text' from 'django.utils.encoding' (/usr/local/lib/python3.9/dist-packages/django/utils/encoding.py)
I already run pip install -U postorius to update that (looks okay to me):
Requirement already satisfied: postorius in /usr/local/lib/python3.9/dist-packages (1.3.8) Collecting postorius Using cached postorius-1.3.8-py3-none-any.whl Downloading postorius-1.3.7.tar.gz (3.2 MB) Requirement already satisfied: django-mailman3>=1.3.8 in /usr/local/lib/python3.9/dist-packages (from postorius) (1.3.9) Requirement already satisfied: readme-renderer[md] in /usr/lib/python3/dist-packages (from postorius) (24.0) Requirement already satisfied: django<4.2,>=3.2 in /usr/local/lib/python3.9/dist-packages (from postorius) (4.1.9) Requirement already satisfied: mailmanclient>=3.3.3 in /usr/local/lib/python3.9/dist-packages (from postorius) (3.3.5) Requirement already satisfied: sqlparse>=0.2.2 in /usr/lib/python3/dist-packages (from django<4.2,>=3.2->postorius) (0.4.1) Requirement already satisfied: asgiref<4,>=3.5.2 in /usr/local/lib/python3.9/dist-packages (from django<4.2,>=3.2->postorius) (3.7.2) Requirement already satisfied: typing-extensions>=4 in /usr/local/lib/python3.9/dist-packages (from asgiref<4,>=3.5.2->django<4.2,>=3.2->postorius) (4.6.3) Requirement already satisfied: django-allauth in /usr/lib/python3/dist-packages (from django-mailman3>=1.3.8->postorius) (0.44.0) Requirement already satisfied: pytz in /usr/lib/python3/dist-packages (from django-mailman3>=1.3.8->postorius) (2021.1) Requirement already satisfied: django-gravatar2>=1.0.6 in /usr/lib/python3/dist-packages (from django-mailman3>=1.3.8->postorius) (1.4.4) Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from mailmanclient>=3.3.3->postorius) (2.25.1) Requirement already satisfied: cmarkgfm>=0.2.0 in /usr/lib/python3/dist-packages (from readme-renderer[md]->postorius) (0.4.2)
dpkg -l | grep django ii python3-django 2:2.2.28-1~deb11u2 all High-level Python web development framework ii python3-django-allauth 0.44.0+ds-1+deb11u1 all Django app for local and social authentication (Python 3 version) ii python3-django-appconf 1.0.3-1 all helper class handling configuration defaults of apps - Python 3.x ii python3-django-compressor 2.4-2 all Compresses linked, inline JS or CSS into single cached files - Python 3.x ii python3-django-extensions 3.0.3-3 all Useful extensions for Django projects (Python 3 version) ii python3-django-filters 2.4.0-1 all filter Django QuerySets based on user selections ii python3-django-gravatar2 1.4.4-2 all Python3 library that provides essential Gravatar support ii python3-django-guardian 2.0.0-2 all per object permissions of django for Python3 ii python3-django-haystack 3.0-1 all modular search for Django (Python version) ii python3-django-hyperkitty 1.3.4-4 all Web user interface to access GNU Mailman3 archives ii python3-django-mailman3 1.3.5-2 all Django library to help interaction with Mailman3 (Python 3 version) ii python3-django-picklefield 3.0.1-1 all Pickled object field for Django (Python3 version) ii python3-django-postorius 1.3.4-2+deb11u1 all Web user interface to access GNU Mailman3 ii python3-django-q 1.2.1-1 all Django multiprocessing distributed task queue (Python 3 version) ii python3-djangorestframework 3.12.1-1 all Web APIs for Django, made easy for Python3
debian 11 (bullseye)
Looks to be a problem between django 4 and /usr/local/lib/python3.9/dist-packages/django/utils/encoding.py ?
Please can anybody help?
On 9/30/23 15:23, Wikinaut wrote:
After having changed and re-changed "only" the line in /etc/mailman3/mailman-web.py (which looks to be only a "blinder" and the real problem could be a different one)
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost/mailman3/'
This should not include the mailman3/ path. If http://localhost/mailman3/ gets to Postorius, it should just be 'http://localhost/'.
the web access via Postorius is not working any more and throws "Error 500"
My logfile /var/log/mailman3/web/mailman-web.log shows now
...> from django.utils.encoding import force_bytes, force_str, force_text
ImportError: cannot import name 'force_text' from 'django.utils.encoding' (/usr/local/lib/python3.9/dist-packages/django/utils/encoding.py)
I already run pip install -U postorius to update that (looks okay to me):
Requirement already satisfied: postorius in /usr/local/lib/python3.9/dist-packages (1.3.8) ... Requirement already satisfied: django<4.2,>=3.2 in /usr/local/lib/python3.9/dist-packages (from postorius) (4.1.9) ... dpkg -l | grep django ii python3-django 2:2.2.28-1~deb11u2 all High-level Python web development framework
Something is confused here. Your Django version from dpkg is 2.2.28, bit
from your above pip install, it is 4.1.9. I'm not sure haw the Debian
package installs, but probably in a venv.
ii python3-django-q 1.2.1-1 all Django multiprocessing distributed task queue (Python 3 version)
I don't know what the django-q version in the venv is, but Django >=4.0 requires django-q >=1.3.0
If The Debian backage includes Django>=4.0 and django-q<1.3.0, it's a Debian packaging issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hmmm, regarding POSTORIUS_TEMPLATE_BASE_URL , just as a remark, 'http://localhost/mailman3/', this was the installed default value.
Regarding the django issue: I do not remember to have triggered an update or so when suddently the error popped up, I did the pip install -U after that.
Do you have a recommendation for me, a hint, how to fix the problem without losing the database and settings of my mailing lists? It is so strange to see the error just right now (while only concentrating on the "template problem"); mailman3 worked for a long time.
I could not fix the problem, apparently there is a mismatch somewhere now.
sudo pip uninstall django-q Found existing installation: django-q 1.3.9 Uninstalling django-q-1.3.9: Would remove: /usr/local/lib/python3.9/dist-packages/CHANGELOG.md /usr/local/lib/python3.9/dist-packages/django_q-1.3.9.dist-info/* /usr/local/lib/python3.9/dist-packages/django_q/* Proceed (y/n)? n
whereas dpkg -l | grep django-q:
ii python3-django-q 1.2.1-1
/usr/lib/python3/dist-info: 4 drwxr-xr-x 8 root root 4096 21. Jun 2022 django_q 4 drwxr-xr-x 2 root root 4096 21. Jun 2022 django_q-1.2.1.egg-info sudo apt remove python3-django-q the following packet will be REMOVED: mailman3-full mailman3-web python3-django-hyperkitty python3-django-q Continue [Y/n] n
What to do? I don't want to remove mailman3, I just want to bring the 1.2.1-1 to 1.3.9...
On Sun, Oct 1, 2023 at 10:46 AM Wikinaut <mail@tgries.de> wrote:
I could not fix the problem, apparently there is a mismatch somewhere now.
sudo pip uninstall django-q Found existing installation: django-q 1.3.9 Uninstalling django-q-1.3.9: Would remove: /usr/local/lib/python3.9/dist-packages/CHANGELOG.md /usr/local/lib/python3.9/dist-packages/django_q-1.3.9.dist-info/* /usr/local/lib/python3.9/dist-packages/django_q/* Proceed (y/n)? n
whereas dpkg -l | grep django-q:
ii python3-django-q 1.2.1-1
/usr/lib/python3/dist-info: 4 drwxr-xr-x 8 root root 4096 21. Jun 2022 django_q 4 drwxr-xr-x 2 root root 4096 21. Jun 2022 django_q-1.2.1.egg-info
sudo apt remove python3-django-q the following packet will be REMOVED: mailman3-full mailman3-web python3-django-hyperkitty python3-django-q Continue [Y/n] n
What to do? I don't want to remove mailman3, I just want to bring the 1.2.1-1 to 1.3.9...
Maybe you can make your life a little more bearable by abandoning the use of Debian packages and embracing the virtualenv install documented at https://docs.list.org/en/latest/install/virtualenv.html#virtualenv-install and we'll find it easier to help you.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Maybe you can make your life a little more bearable by abandoning the use of Debian packages and embracing the virtualenv....
I fully understand. Need to look up how to smoothly backup the mailman3 settings and list databases in order to make that move with losing stuff... (kinda "mailman3 backup all" and mailman3 restore all" :-)
On Sun, Oct 1, 2023 at 1:41 PM Wikinaut <mail@tgries.de> wrote:
Maybe you can make your life a little more bearable by abandoning the use of Debian packages and embracing the virtualenv....
I fully understand. Need to look up how to smoothly backup the mailman3 settings and list databases in order to make that move with losing stuff... (kinda "mailman3 backup all" and mailman3 restore all" :-)
You do not really need to change much. The /etc/mailman3/* are almost usable as it with a little modification to file paths. The databases are also usable as-is. You may need to move a few files to the virtualenv directory tree and change permissions accordingly.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Wikinaut writes:
I fully understand. Need to look up how to smoothly backup the mailman3 settings and list databases in order to make that move with losing stuff... (kinda "mailman3 backup all" and mailman3 restore all" :-)
One piece of generic Python advice: don't mix OS packages and pip (except for Python itself). Always use a venv if you need to use pip for packages that aren't in your OS distro. (Of course it's possible to do so in many cases if you *really* know what you're doing, but it's almost never worth the pain.)
All of the site configuration data should be in /etc/mailman3 (except for a few template files in $var_dir/templates), and all of the domain, list configuration, and archive data is in the RDBMS (usually PostgreSQL or MySQL). The RDMBS should be backed up regularly according to its procedures, but that's completely out of scope for Mailman.[1] The RDBMS should "just work" with exactly the same Mailman-side configuration as Debian.[2]
As I think Odhiambo mentioned, the Mailman side of the site configuration also does not change (for example, the RDBMS URL and authentication). What you do want to change is the file system configuration. Except for /etc/mailman3, it should be completely different -- as with pip'ing packages, you don't want to share files across a local venv and the OS version of the same package. It just leads to pain.
I think among the developers the preferred root = /opt/mailman (or
/opt/mailman3) for Mailman's directory hierarchy. This will be
populated with subdirectories when you run mailman info (or start
Mailman) the first time. The directories relevant to configuration
are all under $root/var, and you inform Mailman about this by setting
var_dir in mailman.cfg. If there are other file and directory
configurations in mailman.cfg, they need to changed from the Debian
values (probably /var/lib/mailman3/...). You may do fine by simply
deleting all other directory configurations and let Mailman decide
where to put them relative to the var_dir,
You will need to fix the setting for STATIC_ROOT (I think that's the name, if not search for "static") in settings.py, and maybe in mailman-web.cfg. While it's not necessary (and may be a little inconvenient), I recommend changing the root for Mailman's log files from the Debian value (probably /var/log/mailman3) to the Mailman default of $root/var/logs, and similarly for the Django logs in settings.py.
Non-Mailman settings
I think you will need to change the settings for your MTA (usually Postfix or Exim4) so that it will recognize Mailman lists and route them to Mailman correctly. There will probably be mailman-related files in /etc/$MTA/conf.d (or something like that). You can edit them directly, but I recommend changing the name (paranoia so that dpkg doesn't overwrite them).
The only thing I can think of for the webserver (Apache?) is the location of the static directory. I think Debian uses uwsgi by default so you may need to adjust its configuration too.
I think you need to fix logrotate's configuration in /etc/logrotate.d. Not sure how many mailman-related config files are in there.
I think you mentioned systemd .service files. If so that doesn't make
sense to me -- Debian should provide those. You need to make sure
those are disabled or deleted so that Debian Mailman services don't
start on reboot. This is automatic if you delete the Debian packages,
of course, but not for any files you created yourself. Probably the
Debian .service files will work, but you need to make sure they refer
to the venv Mailman (using the full path to the mailman utility in
the venv's bin directory will work, and similarly for mailman-web).
systemd does not configure the /opt hierarchy in it search path by
default. I put the mailman, qcluster, and gunicorn (if you use that
instead of uwsgi) .service files in /usr/local/lib/systemd/system, and
that works for me.
Flotsam and jetsam
If you have run Mailman with any success at all, you may find some
stuff in Debian's $var_dir/queue/* directories or in its
$var_dir/lists/* directories. The queue/* directories contain
incoming messages at various stages of processing, and the lists/*
directories contain files containing multiple messages that are
scheduled to be sent out as digests. You can read the queue files
with mailman qfile /path/to/queue/file.pck, and the digests are just
text files. That way you can check if there's anything important in
them. If not, you can just delete them. If there is, you can just
copy them to the same place in the venv $var_dir as they were in
Debian's $var_dir, but you may want to ask here first before doing
that. It's best to let Mailman run until all the queues are cleared
(except maybe bad and shunt, but if those have any files in them you
need to intervene anyway).
Afterword
I'm sure this sounds complicated, but in fact it's a pretty short list. When you get there we'll help you.
Footnotes: [1] But not for this list. You may as well ask here, somebody probably will tell you the basics quickly. But for troubleshooting, go the the RDBMS's own channels.
[2] In fact, it is possible to keep the Debian installation, copy the the mailman.cfg somewhere, edit it, stop Debian Mailman, and start up the venv Mailman. I don't recommend this -- the point is that the venv and the databases are that isolated from the rest of the system.
participants (4)
-
Mark Sapiro -
Odhiambo Washington -
Stephen J. Turnbull -
Wikinaut