*<< I sent this to the mailman3 issues on gitlab and Mark replied but asked me to move to this list, so I copied everything and inserted responses. I hope this makes sense and someone can help me.>>*
After a lengthy power failure I had to reboot a machine that has been running for several years. the WEB-UI does not seem to be able to see mailman-core. I have looked for log entries that might explain what the issue is and I am stumped to even find logs that will update when I get an error on the browser. Mailman is definitely running, postgres is running and mailman web is running. if I go to the lists page I see lists, if I try to access a specific list I get Server error: An error occurred while processing your request.
I looked in /var/logs/mailman /opt/mailman/web/mailmanweb
All I can find is:
django.urls.exceptions.NoReverseMatch: Reverse for 'list_unsubscription_requests' not found. 'list_unsubscription_requests' is not a valid view function or pattern name.16688 ERROR 2023-09-12 13:56:58,430 1760 django.request Internal Server Error: /mailman3/lists/goodstein.goodstein.org/ File "/opt/mailman/venv/lib64/python3.8/site-packages/haystack/admin.py", line 8, in <module> from django.utils.translation import ungettextImportError: cannot import name 'ungettext' from 'django.utils.translation' (/opt/mailman/venv/lib64/python3.8/site-packages/django/utils/translation/__init__.py)
Also I have to run mailman from "mailman start" because the systemctl returns this:
● mailman3.service - GNU Mailing List Manager Loaded: loaded (/etc/systemd/system/mailman3.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2023-09-12 10:06:31 EDT; 13s ago Process: 31159 ExecStart=/opt/mailman/venv/bin/mailman start (code=exited, status=1/FAILURE) CPU: 806msSep 12 10:06:31 mail mailman[31159]: Error in atexit._run_exitfuncs:Sep 12 10:06:31 mail mailman[31159]: Traceback (most recent call last):Sep 12 10:06:31 mail mailman[31159]: File "/usr/lib64/python3.8/logging/__init__.py", line 2126, in shutdownSep 12 10:06:31 mail mailman[31159]: h.flush()Sep 12 10:06:31 mail mailman[31159]: File "/opt/mailman/venv/lib64/python3.8/site-packages/mailman/core/logging.py", line 67, in flushSep 12 10:06:31 mail mailman[31159]: if self._stream:Sep 12 10:06:31 mail mailman[31159]: AttributeError: 'ReopenableFileHandler' object has no attribute '_stream'Sep 12 10:06:31 mail systemd[1]: mailman3.service: Control process exited, code=exited, status=1/FAILURESep 12 10:06:31 mail systemd[1]: mailman3.service: Failed with result 'exit-code'.Sep 12 10:06:31 mail systemd[1]: Failed to start GNU Mailing List Manager.
Please tell me where to start looking or adjusting.
Reply from Mark Sapiro,* Edited to include responses:*
Mark Sapiro <https://gitlab.com/msapiro>@msapiro <https://gitlab.com/msapiro>· 1 day ago <https://gitlab.com/mailman/mailman-web/-/issues/23#note_1557601916> Owner Add reaction More actions
The mailman-users@mailman3.org list is probably a more appropriate place for issues like this.
django.urls.exceptions.NoReverseMatch: Reverse for 'list_unsubscription_requests' not found. 'list_unsubscription_requests' is not a valid view function or pattern name. 16688 ERROR 2023-09-12 13:56:58,430 1760 django.request Internal Server Error: /mailman3/lists/ goodstein.goodstein.org/
/opt/mailman/venv/lib64/python3.8/site-packages/postorius/urls.py should contain
re_path( r'^unsubscription_requests$',list_views.list_unsubscription_requests, name='list_unsubscription_requests', ),
Y
*Yes this exists*
and /opt/mailman/venv/lib64/python3.8/site-packages/postorius/views/list.py should define list_unsubscription_requests.
*Yes there is a method by this name defined*
File "/opt/mailman/venv/lib64/python3.8/site-packages/haystack/admin.py", line 8, in from django.utils.translation import ungettext ImportError: cannot import name 'ungettext' from 'django.utils.translation' (/opt/mailman/venv/lib64/python3.8/site-packages/django/utils/translation/ *init*.py)
See https://docs.djangoproject.com/en/4.2/internals/deprecation/#deprecation-rem...
ungettext() is removed from django>=4.0
Sep 12 10:06:31 mail mailman[31159]: AttributeError: 'ReopenableFileHandler' object has no attribute '_stream'
I don't understand this one offhand, but what is the content of /etc/systemd/system/mailman3.service, in particular the settings for User, Group and ExecStart?
*The service looks right:*
1 [Unit] 2 Description=GNU Mailing List Manager 3 After=syslog.target network.target postgresql.service 4 5 [Service] 6 Type=forking 7 PIDFile=/opt/mailman/mm/var/master.pid 8 User=mailman 9 Group=mailman 10 ExecStart=/opt/mailman/venv/bin/mailman start 11 ExecReload=/opt/mailman/venv/bin/mailman restart 12 ExecStop=/opt/mailman/venv/bin/mailman stop 13 14 [Install] 15 WantedBy=multi-user.target
*It seems I may have two versions of the mailman script:*
mail:/opt/mailman # which mailman /usr/bin/mailman mail:/opt/mailman # ls -al /usr/bin/mailman -rwxr-xr-x 1 root root 382 May 10 2021 /usr/bin/mailman mail:/opt/mailman # ls -al /opt/mailman/venv/bin/mailman -rwxr-xr-x 1 mailman mailman 231 Sep 12 10:02 /opt/mailman/venv/bin/mailman
Somehow, your installation is corrupted. I don't know how
*I need to know what to remove, how to correctly update, etc., That said the two scripts appear to be very similar:*
mail:/opt/mailman # head /usr/bin/mailman
#!/usr/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'mailman==3.3.4','console_scripts','mailman' __requires__ = 'mailman==3.3.4' import re import sys from pkg_resources import load_entry_point
if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(
mail:/opt/mailman # head /opt/mailman/venv/bin/mailman #!/opt/mailman/venv/bin/python3 # -*- coding: utf-8 -*- import re import sys from mailman.bin.mailman import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(main()) mail:/opt/mailman #
*Given the two years since the last reboot I also tried following * https://docs.mailman3.org/en/latest/upgrade-guide.html *to make sure I was running the latest*
pip install -U mailman postorius django-mailman3 hyperkitty mailman-web
On Sat, Sep 16, 2023 at 5:49 PM Paul Rubin <paulrubin37@gmail.com> wrote:
*<< I sent this to the mailman3 issues on gitlab and Mark replied but asked me to move to this list, so I copied everything and inserted responses. I hope this makes sense and someone can help me.>>*
After a lengthy power failure I had to reboot a machine that has been running for several years. the WEB-UI does not seem to be able to see mailman-core. I have looked for log entries that might explain what the issue is and I am stumped to even find logs that will update when I get an error on the browser. Mailman is definitely running, postgres is running and mailman web is running. if I go to the lists page I see lists, if I try to access a specific list I get Server error: An error occurred while processing your request.
I looked in /var/logs/mailman /opt/mailman/web/mailmanweb
All I can find is:
django.urls.exceptions.NoReverseMatch: Reverse for 'list_unsubscription_requests' not found. 'list_unsubscription_requests' is not a valid view function or pattern name.16688 ERROR 2023-09-12 13:56:58,430 1760 django.request Internal Server Error: /mailman3/lists/goodstein.goodstein.org/ File "/opt/mailman/venv/lib64/python3.8/site-packages/haystack/admin.py", line 8, in <module> from django.utils.translation import ungettextImportError: cannot import name 'ungettext' from 'django.utils.translation'
(/opt/mailman/venv/lib64/python3.8/site-packages/django/utils/translation/__init__.py)
Also I have to run mailman from "mailman start" because the systemctl returns this:
● mailman3.service - GNU Mailing List Manager Loaded: loaded (/etc/systemd/system/mailman3.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2023-09-12 10:06:31 EDT; 13s ago Process: 31159 ExecStart=/opt/mailman/venv/bin/mailman start (code=exited, status=1/FAILURE) CPU: 806msSep 12 10:06:31 mail mailman[31159]: Error in atexit._run_exitfuncs:Sep 12 10:06:31 mail mailman[31159]: Traceback (most recent call last):Sep 12 10:06:31 mail mailman[31159]: File "/usr/lib64/python3.8/logging/__init__.py", line 2126, in shutdownSep 12 10:06:31 mail mailman[31159]: h.flush()Sep 12 10:06:31 mail mailman[31159]: File "/opt/mailman/venv/lib64/python3.8/site-packages/mailman/core/logging.py", line 67, in flushSep 12 10:06:31 mail mailman[31159]: if self._stream:Sep 12 10:06:31 mail mailman[31159]: AttributeError: 'ReopenableFileHandler' object has no attribute '_stream'Sep 12 10:06:31 mail systemd[1]: mailman3.service: Control process exited, code=exited, status=1/FAILURESep 12 10:06:31 mail systemd[1]: mailman3.service: Failed with result 'exit-code'.Sep 12 10:06:31 mail systemd[1]: Failed to start GNU Mailing List Manager.
Please tell me where to start looking or adjusting.
Reply from Mark Sapiro,* Edited to include responses:*
Mark Sapiro <https://gitlab.com/msapiro>@msapiro <https://gitlab.com/msapiro>· 1 day ago <https://gitlab.com/mailman/mailman-web/-/issues/23#note_1557601916> Owner Add reaction More actions
The mailman-users@mailman3.org list is probably a more appropriate place for issues like this.
django.urls.exceptions.NoReverseMatch: Reverse for 'list_unsubscription_requests' not found. 'list_unsubscription_requests' is not a valid view function or pattern name. 16688 ERROR 2023-09-12 13:56:58,430 1760 django.request Internal Server Error: /mailman3/lists/ goodstein.goodstein.org/
/opt/mailman/venv/lib64/python3.8/site-packages/postorius/urls.py should contain
re_path( r'^unsubscription_requests$',list_views.list_unsubscription_requests, name='list_unsubscription_requests', ),
Y
*Yes this exists*
and /opt/mailman/venv/lib64/python3.8/site-packages/postorius/views/list.py should define list_unsubscription_requests.
*Yes there is a method by this name defined*
File "/opt/mailman/venv/lib64/python3.8/site-packages/haystack/admin.py", line 8, in from django.utils.translation import ungettext ImportError: cannot import name 'ungettext' from 'django.utils.translation' (/opt/mailman/venv/lib64/python3.8/site-packages/django/utils/translation/ *init*.py)
See
https://docs.djangoproject.com/en/4.2/internals/deprecation/#deprecation-rem...
ungettext() is removed from django>=4.0
Sep 12 10:06:31 mail mailman[31159]: AttributeError: 'ReopenableFileHandler' object has no attribute '_stream'
I don't understand this one offhand, but what is the content of /etc/systemd/system/mailman3.service, in particular the settings for User, Group and ExecStart?
*The service looks right:*
1 [Unit] 2 Description=GNU Mailing List Manager 3 After=syslog.target network.target postgresql.service 4 5 [Service] 6 Type=forking 7 PIDFile=/opt/mailman/mm/var/master.pid 8 User=mailman 9 Group=mailman 10 ExecStart=/opt/mailman/venv/bin/mailman start 11 ExecReload=/opt/mailman/venv/bin/mailman restart 12 ExecStop=/opt/mailman/venv/bin/mailman stop 13 14 [Install] 15 WantedBy=multi-user.target
*It seems I may have two versions of the mailman script:*
mail:/opt/mailman # which mailman /usr/bin/mailman mail:/opt/mailman # ls -al /usr/bin/mailman -rwxr-xr-x 1 root root 382 May 10 2021 /usr/bin/mailman mail:/opt/mailman # ls -al /opt/mailman/venv/bin/mailman -rwxr-xr-x 1 mailman mailman 231 Sep 12 10:02 /opt/mailman/venv/bin/mailman
Somehow, your installation is corrupted. I don't know how
*I need to know what to remove, how to correctly update, etc., That said the two scripts appear to be very similar:*
mail:/opt/mailman # head /usr/bin/mailman
#!/usr/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'mailman==3.3.4','console_scripts','mailman' __requires__ = 'mailman==3.3.4' import re import sys from pkg_resources import load_entry_point
if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(
mail:/opt/mailman # head /opt/mailman/venv/bin/mailman #!/opt/mailman/venv/bin/python3 # -*- coding: utf-8 -*- import re import sys from mailman.bin.mailman import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(main()) mail:/opt/mailman #
*Given the two years since the last reboot I also tried following * https://docs.mailman3.org/en/latest/upgrade-guide.html *to make sure I was running the latest*
pip install -U mailman postorius django-mailman3 hyperkitty mailman-web
Why don't you try and do a fresh virtualenv install and move your configs into it?
- cd /opt
- mv mailman _mailman.BAK
- Create a backup of the database.
Then follow this:
https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-insta...,
but do not create the database as it already exists.
You also do not need to run any database migrations, so you can actually
skip those
You just need to ensure that
mailman infosays that its configuration files are in /etc/mailman3/ Create the necessary directories as per the above installation instructions.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Paul Rubin writes:
'list_unsubscription_requests' not found.
ungettextImportError: cannot import name 'ungettext' from 'django.utils.translation'
self._stream:Sep 12 10:06:31 mail mailman[31159]: AttributeError: 'ReopenableFileHandler' object has no attribute '_stream'Sep 12
Please tell me where to start looking or adjusting.
At this point it's not a matter of "adjusting". There are too many errors in too many unrelated places: Mailman, Postorius, and Django. It appears your Mailman installation has been quite corrupted. My guess is in the process of recovering the host you applied OS upgrades to Python and/or reinstalled Mailman components, and the versions are now very much out of sync. Another possibility is that files were corrupted by the crash.
As Odhiambo Washington suggests in a parallel post, your best bet is to preserve your configuration files, Postgresql databases, and queue files (if any), wipe your Mailman installation and reinstall from scratch. I'd back up the Postgres database, of course, but Mailman should just "reattach" to it, applying any migrations automatically. I forget whether you need to run "mailman-web migrate" for Django but I'm pretty sure it's idempotent (you did back up Postgres, right?)
Steve
On 9/16/23 5:29 AM, Paul Rubin wrote:
*I need to know what to remove, how to correctly update, etc., That said the two scripts appear to be very similar:*
mail:/opt/mailman # head /usr/bin/mailman
#!/usr/bin/python3 # EASY-INSTALL-ENTRY-SCRIPT: 'mailman==3.3.4','console_scripts','mailman' __requires__ = 'mailman==3.3.4' import re import sys from pkg_resources import load_entry_point
if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(
This is no good because it uses the system Python and won't have the venv's site-packages in its path.
mail:/opt/mailman # head /opt/mailman/venv/bin/mailman #!/opt/mailman/venv/bin/python3 # -*- coding: utf-8 -*- import re import sys from mailman.bin.mailman import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(main()) mail:/opt/mailman #
This one is good because it uses the venv python3 and thus uses site-packages from the venv.
*Given the two years since the last reboot I also tried following * https://docs.mailman3.org/en/latest/upgrade-guide.html *to make sure I was running the latest*
pip install -U mailman postorius django-mailman3 hyperkitty mailman-web
Thin may be part of the issue. The above may not upgrade dependencies as
needed. You may need the --upgrade-strategy eager option.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I went through the install process in the provided link when I tried to start the service I keep getting errors about:
Sep 27 19:17:15 mail mailman[1771]: ImportError: cannot import name 'path' from 'importlib_resources' (/opt/mailman/venv/lib64/python3.8/site-packages/importlib_resources/__init__.py)
I verified the python version:
(venv) mailman@mail:~> python -V Python 3.8.10
Any suggestions?
On 9/27/23 16:29, paulrubin37@gmail.com wrote:
I went through the install process in the provided link when I tried to start the service I keep getting errors about:
Sep 27 19:17:15 mail mailman[1771]: ImportError: cannot import name 'path' from 'importlib_resources' (/opt/mailman/venv/lib64/python3.8/site-packages/importlib_resources/__init__.py)
I verified the python version:
(venv) mailman@mail:~> python -V Python 3.8.10
See https://gitlab.com/mailman/mailman/-/issues/1093
The easiest fix is in your venv do
pip install importlib_resources\<6.0
Mailman core <=3.3.8 is incompatible with importlib_resources>=6.0
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Ok, I installed the specific version:
(venv) mailman@mail:~> pip install importlib_resources\<6.0 Collecting importlib_resources<6.0 Obtaining dependency information for importlib_resources<6.0 from https://files.pythonhosted.org/packages/7a/68/bd9dd6bbf06772c7accce77d0354d7... Downloading importlib_resources-5.13.0-py3-none-any.whl.metadata (4.2 kB) Requirement already satisfied: zipp>=3.1.0 in ./venv/lib/python3.8/site-packages (from importlib_resources<6.0) (3.17.0) Downloading importlib_resources-5.13.0-py3-none-any.whl (32 kB) Installing collected packages: importlib_resources Attempting uninstall: importlib_resources Found existing installation: importlib-resources 6.1.0 Uninstalling importlib-resources-6.1.0: Successfully uninstalled importlib-resources-6.1.0 Successfully installed importlib_resources-5.13.0
Then I tried mailman info, that errored with
pkg_resources.DistributionNotFound: The 'flufl.lock>=5.1' distribution was not found and is required by mailman
So I tried installing that (venv) mailman@mail:~> pip install flufl.lock Requirement already satisfied: flufl.lock in ./venv/lib/python3.8/site-packages (8.0.2) Requirement already satisfied: atpublic in ./venv/lib/python3.8/site-packages (from flufl.lock) (4.0) Requirement already satisfied: psutil in ./venv/lib/python3.8/site-packages (from flufl.lock) (5.9.5)
The error did not change
Suggestions?
On 9/28/23 16:15, paulrubin37@gmail.com wrote:
Then I tried mailman info, that errored with
pkg_resources.DistributionNotFound: The 'flufl.lock>=5.1' distribution was not found and is required by mailman
So I tried installing that (venv) mailman@mail:~> pip install flufl.lock Requirement already satisfied: flufl.lock in ./venv/lib/python3.8/site-packages (8.0.2) Requirement already satisfied: atpublic in ./venv/lib/python3.8/site-packages (from flufl.lock) (4.0) Requirement already satisfied: psutil in ./venv/lib/python3.8/site-packages (from flufl.lock) (5.9.5)
The error did not change
This is a packaging glitch with PyPI. At one point a couple of flufl.* packages were named flufl_*. I forget exactly how to fix this, but try in your venv
pip install -U --force-reinstall flufl.bounce flufl.i18n flufl.lock
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 9/28/23 20:25, Mark Sapiro wrote:
This is a packaging glitch with PyPI. At one point a couple of flufl.* packages were named flufl_*. I forget exactly how to fix this, but try in your venv
pip install -U --force-reinstall flufl.bounce flufl.i18n flufl.lock
https://gitlab.com/mailman/mailman/-/issues/1085 and/or: https://mail.python.org/archives/list/mailman-developers@python.org/thread/R... may be relevant.
Nelson
On 9/29/23 06:00, Nelson Strother wrote:
This one is relevant. I knew it was there, but couldn't find it to point to yesterday. Thank you.
and/or: https://mail.python.org/archives/list/mailman-developers@python.org/thread/R...
This is about an older issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Does this make any sense to you?
ERROR: Could not find a version that satisfies the requirement flufl.bounce (from versions: none) ERROR: No matching distribution found for flufl.bounce
On 9/29/23 08:52, paulrubin37@gmail.com wrote:
Does this make any sense to you?
ERROR: Could not find a version that satisfies the requirement flufl.bounce (from versions: none) ERROR: No matching distribution found for flufl.bounce
was that in response to
pip install -U --force-reinstall flufl.bounce flufl.i18n flufl.lock
If so, I don't understand. https://pypi.org/project/flufl.bounce/ https://pypi.org/project/flufl.i18n/ and https://pypi.org/project/flufl.lock/ all exist.
However, I now suspect the issue is somewhat different. See https://gitlab.com/mailman/mailman/-/issues/1085#note_1442207929 although that refers to $VENV_ROOT/lib/$PYVERSION/site-packages/mailman-3.3.8.egg-info/requires.txt and in your case, that might instead be $VENV_ROOT/lib/$PYVERSION/site-packages/mailman-3.3.8.dist-info/METADATA
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Ok, I edited request.txt for -lock and -in8 and got:
(venv) mailman@mail:~> mailman info GNU Mailman 3.3.8 (Tom Sawyer) Python 3.8.10 (default, May 05 2021, 15:36:36) [GCC] config file: /etc/mailman3/mailman.cfg db url: postgresql://mailman:xxxxx@localhost/mailman devmode: DISABLED REST root url: http://localhost:8001/3.1/ REST credentials: restadmin:restpass (venv) mailman@mail:~>
That got me:
mail:/opt # systemctl start mailman3 mail:/opt # systemctl status mailman3 ● mailman3.service - GNU Mailing List Manager Loaded: loaded (/etc/systemd/system/mailman3.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2023-09-29 15:36:13 EDT; 4s ago Process: 20268 ExecStart=/opt/mailman/venv/bin/mailman start (code=exited, status=0/SUCCESS) Main PID: 20271 (python3) Tasks: 15 (limit: 4622) CPU: 16.548s CGroup: /system.slice/mailman3.service ├─20271 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/master -C /etc/mailman3/mailman.cfg ├─20277 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=archive:0:1 ├─20278 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=bounces:0:1 ├─20279 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=command:0:1 ├─20280 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=in:0:1 ├─20281 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=lmtp:0:1 ├─20282 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=nntp:0:1 ├─20283 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=out:0:1 ├─20284 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=pipeline:0:1 ├─20285 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1 ├─20286 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=retry:0:1 ├─20287 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=task:0:1 ├─20288 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=virgin:0:1 ├─20289 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=digest:0:1 └─20306 /opt/mailman/venv/bin/python3 /opt/mailman/venv/bin/mailman -C /etc/mailman3/mailman.cfg gatenews
Sep 29 15:36:11 mail systemd[1]: Starting GNU Mailing List Manager... Sep 29 15:36:12 mail mailman[20268]: Starting Mailman's master runner Sep 29 15:36:12 mail mailman[20268]: Generating MTA alias maps Sep 29 15:36:12 mail systemd[1]: mailman3.service: Can't open PID file /opt/mailman/mm/var/master.pid (yet?) after start: Operation not permitted Sep 29 15:36:13 mail systemd[1]: Started GNU Mailing List Manager.
YEAH! progress.
When I tried to use mailman-web I get this error:
django.core.exceptions.ImproperlyConfigured: allauth.account.middleware.AccountMiddleware must be added to settings.MIDDLEWARE
Which config file is this and what do I need to add?
On 9/29/23 12:53, paulrubin37@gmail.com wrote:
When I tried to use mailman-web I get this error:
django.core.exceptions.ImproperlyConfigured: allauth.account.middleware.AccountMiddleware must be added to settings.MIDDLEWARE
This is because you have django-allauth>=0.56.
If your django-mailman3 version is <=1.3.9 you can downgrade django-allauth with
pip install django-allauth\<0.56
Or you can find where in your django settings MIDDLEWARE is defined and add
'allauth.account.middleware.AccountMiddleware',
to the list, however if you have a /etc/mailman3/settings.py like the one at https://docs.mailman3.org/en/latest/install/virtualenv.html#initial-configur... the imports from mailman_web.settings.base, you need to add the following to /etc/mailman3/settings.py
x = list(MIDDLEWARE)
x.insert(0, 'allauth.account.middleware.AccountMiddleware')
MIDDLEWARE = x
Note either downgrade django-allauth or update MIDDLEWARE, but not both.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank you , more progress.
Unfortunately, I did a stupid. Given all the error and what not I got the stupid idea to update the box entirely. The venv is gone and I am unable to re-install it:
mail:~ # apt install python3-dev python3-venv sassc lynx Loading repository data... Reading installed packages... 'python3-devel' not found in package names. Trying capabilities. 'python311-devel' providing 'python3-devel' is already installed. 'lynx' is already installed. No update candidate for 'lynx-2.9.0~dev.10-1.9.x86_64'. The highest available version is already installed. 'sassc' is already installed. No update candidate for 'sassc-3.6.2-1.10.x86_64'. The highest available version is already installed. 'python3-venv' not found in package names. Trying capabilities.
I have searched google and not found the correct repo to add to fix this. Is there any chance you know?
On 10/1/23 11:29, paulrubin37@gmail.com wrote:
Thank you , more progress.
Unfortunately, I did a stupid. Given all the error and what not I got the stupid idea to update the box entirely. The venv is gone and I am unable to re-install it:
mail:~ # apt install python3-dev python3-venv sassc lynx ... 'python3-venv' not found in package names. Trying capabilities.
I have searched google and not found the correct repo to add to fix this. Is there any chance you know?
What does apt list python3-v* show?
What does python3 -m venv --help show?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thank you for sticking with me.
mail:~ # apt list python3-v* Retrieving repository 'YaST:Head' metadata .......................................................................................................................................................[done] Building repository 'YaST:Head' cache ............................................................................................................................................................[done] Loading repository data... Reading installed packages...
S | Name | Summary | Type --+---------------------------+----------------------------------------------------------------------------+-------- | python3-vapoursynth | Python interface for VapourSynth | package | python3-virt-bootstrap | System container rootfs creation tool | package | python3-virt-scenario | Tool to create XML guest configuration and prepare the host for a scenario | package | python3-virt-tuner | Virtual Machine definition tuner | package | python3-virtualbox | Python bindings for virtualbox | package | python3-virtualenvwrapper | Enhancements to virtualenv | package | python3-vtk | Python bindings for VTK | package | python3-vtk-openmpi2 | Python bindings for VTK | package | python3-vtk-openmpi3 | Python bindings for VTK | package | python3-vtk-openmpi4 | Python bindings for VTK | package
Note: For an extended search including not yet activated remote resources please use 'zypper
search-packages'.mail:~ # python3 -m venv --help usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps] ENV_DIR [ENV_DIR ...]
Creates virtual Python environments in one or more target directories.
positional arguments: ENV_DIR A directory to create the environment in.
options: -h, --help show this help message and exit --system-site-packages Give the virtual environment access to the system site-packages dir. --symlinks Try to use symlinks rather than copies, when symlinks are not the default for the platform. --copies Try to use copies rather than symlinks, even when symlinks are the default for the platform. --clear Delete the contents of the environment directory if it already exists, before environment creation. --upgrade Upgrade the environment directory to use this version of Python, assuming Python has been upgraded in-place. --without-pip Skips installing or upgrading pip in the virtual environment (pip is bootstrapped by default) --prompt PROMPT Provides an alternative prompt prefix for this environment. --upgrade-deps Upgrade core dependencies: pip setuptools to the latest version in PyPI
Once an environment has been created, you may wish to activate it, e.g. by sourcing an activate script in its bin directory. mail:~ #
paulrubin37@gmail.com writes:
| python3-virtualenvwrapper | Enhancements to virtualenv | package
I suspect this is part of the problem, it's probably blocking any other Debian virtualenv packages as "conflicts".
mail:~ # python3 -m venv --help usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] [--upgrade-deps] ENV_DIR [ENV_DIR ...]
You should be able to use "python3 -m venv $ENV_DIR" to create an appropriate virtual environment, and then
. /path/to/venv/bin/activate
python -m pip --upgrade install pip # NOTE: using the venv python3!
python -m pip install build wheelfollowed by installing mailman and friends in the same way.
If there are complaints about pip itself, I believe "python -m ensure_pip" should do the trick.
Thank you guys, I am inching along. I had to fight my way through install psycopg2 I finally got the path right to get that install to run. It seems however that that did not quite fix me. I see the stack trace, but not a specific error. If I follow the stack trace it is failing on:
51 from psycopg2._psycopg import ( # noqa 52 BINARY, NUMBER, STRING, DATETIME, ROWID, 53 54 Binary, Date, Time, Timestamp, 55 DateFromTicks, TimeFromTicks, TimestampFromTicks,
Which I would have thought this install provided
(venv) mailman@mail:~> pip install psycopg2-binary\<2.9 Collecting psycopg2-binary<2.9 Using cached psycopg2-binary-2.8.6.tar.gz (384 kB) Preparing metadata (setup.py) ... done Building wheels for collected packages: psycopg2-binary Building wheel for psycopg2-binary (setup.py) ... done Created wheel for psycopg2-binary: filename=psycopg2_binary-2.8.6-cp311-cp311-linux_x86_64.whl size=452300 sha256=4b7bb83bb4c093dceb8d8a67c902a1a42fdab559012fea879eabf5239447510a Stored in directory: /opt/mailman/.cache/pip/wheels/52/05/a7/b2e1f6441d09730f2e8abfc40df38aff1c2ef490d3314072a8 Successfully built psycopg2-binary Installing collected packages: psycopg2-binary Successfully installed psycopg2-binary-2.8.6
Here is the full stack trace
(venv) mailman@mail:~> mailman info Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1077, in main with self.make_context(prog_name, args, **extra) as ctx: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 943, in make_context self.parse_args(ctx, args) File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1644, in parse_args rest = super().parse_args(ctx, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1408, in parse_args value, args = param.handle_parse_result(ctx, opts, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result value = self.process_value(ctx, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 2362, in process_value value = self.callback(ctx, self, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/bin/mailman.py", line 95, in initialize_config initialize(value) File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/core/initialize.py", line 229, in initialize initialize_2(propagate_logs=propagate_logs) File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/core/initialize.py", line 187, in initialize_2 config.db = getUtility(IDatabaseFactory, utility_name).create() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/database/factory.py", line 57, in create database.initialize() File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/database/base.py", line 117, in initialize self.engine = create_engine( ^^^^^^^^^^^^^^ File "<string>", line 2, in create_engine File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned return fn(*args, **kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/engine/create.py", line 601, in create_engine dbapi = dbapi_meth(**dbapi_args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi import psycopg2 File "/opt/mailman/venv/lib64/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa SystemError: initialization of _psycopg raised unreported exception
Looking forward to your next reply.
On Tue, Oct 3, 2023 at 3:53 AM <paulrubin37@gmail.com> wrote:
Thank you guys, I am inching along. I had to fight my way through install psycopg2 I finally got the path right to get that install to run. It seems however that that did not quite fix me. I see the stack trace, but not a specific error. If I follow the stack trace it is failing on:
51 from psycopg2._psycopg import ( # noqa 52 BINARY, NUMBER, STRING, DATETIME, ROWID, 53 54 Binary, Date, Time, Timestamp, 55 DateFromTicks, TimeFromTicks, TimestampFromTicks,
Which I would have thought this install provided
(venv) mailman@mail:~> pip install psycopg2-binary\<2.9 Collecting psycopg2-binary<2.9 Using cached psycopg2-binary-2.8.6.tar.gz (384 kB) Preparing metadata (setup.py) ... done Building wheels for collected packages: psycopg2-binary Building wheel for psycopg2-binary (setup.py) ... done Created wheel for psycopg2-binary: filename=psycopg2_binary-2.8.6-cp311-cp311-linux_x86_64.whl size=452300 sha256=4b7bb83bb4c093dceb8d8a67c902a1a42fdab559012fea879eabf5239447510a Stored in directory: /opt/mailman/.cache/pip/wheels/52/05/a7/b2e1f6441d09730f2e8abfc40df38aff1c2ef490d3314072a8 Successfully built psycopg2-binary Installing collected packages: psycopg2-binary Successfully installed psycopg2-binary-2.8.6
Here is the full stack trace
(venv) mailman@mail:~> mailman info Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1077, in main with self.make_context(prog_name, args, **extra) as ctx: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 943, in make_context self.parse_args(ctx, args) File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1644, in parse_args rest = super().parse_args(ctx, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1408, in parse_args value, args = param.handle_parse_result(ctx, opts, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result value = self.process_value(ctx, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 2362, in process_value value = self.callback(ctx, self, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/bin/mailman.py", line 95, in initialize_config initialize(value) File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/core/initialize.py", line 229, in initialize initialize_2(propagate_logs=propagate_logs) File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/core/initialize.py", line 187, in initialize_2 config.db = getUtility(IDatabaseFactory, utility_name).create() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/database/factory.py", line 57, in create database.initialize() File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/database/base.py", line 117, in initialize self.engine = create_engine( ^^^^^^^^^^^^^^ File "<string>", line 2, in create_engine File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned return fn(*args, **kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/engine/create.py", line 601, in create_engine dbapi = dbapi_meth(**dbapi_args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi import psycopg2 File "/opt/mailman/venv/lib64/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa SystemError: initialization of _psycopg raised unreported exception
Looking forward to your next reply.
At this point, I personally would simply move this 'dirty' venv aside and create a new one, then bring in my /opt/mailman/{mm,var,web} to the new venv.
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]
Odhiambo Washington writes:
On Tue, Oct 3, 2023 at 3:53?AM <paulrubin37@gmail.com> wrote:
Thank you guys, I am inching along. I had to fight my way through install psycopg2 I finally got the path right to get that install to run. It seems however that that did not quite fix me. I see the stack trace, but not a specific error. If I follow the stack trace it is failing on:
[omitted]
File "/opt/mailman/venv/lib64/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa SystemError: initialization of _psycopg raised unreported exception
This is really bad news. The only way to debug this is to become a psycopg developer. My guess is your venv is still polluted with Debian stuff.
At this point, I personally would simply move this 'dirty' venv aside and create a new one, then bring in my /opt/mailman/{mm,var,web} to the new venv.
I agree with Odhiambo: blow this away and try again.
Steve
Sorry folks. I am back again. I completely wiped out /opt/mailman. I went into the os package manager and removed all their versions just to make sure that did not interfere. I started the instructions again I followed the basic directions plus all our adjustments from the previous mail chain like manually install import_resources
Note there were zero issues this time with venv or pip.
python3 -m venv venv vi .bashrc exit pip install wheel mailman psycopg2-binary\<2.9 vi /etc/mailman3/mailman.cfg mailman info <-- had the path error pip install "importlib_resources<6.0" mailman info <-- has the error below.
I checked postgresql: (venv) mailman@mail:~> psql psql (16.0, server 13.12) Type "help" for help.
mailman=> \dt List of relations Schema | Name | Type | Owner --------+--------------------+-------+--------- public | _request | table | mailman public | acceptablealias | table | mailman public | address | table | mailman public | alembic_version | table | mailman public | autoresponserecord | table | mailman public | ban | table | mailman public | bounceevent | table | mailman public | contentfilter | table | mailman public | domain | table | mailman public | domain_owner | table | mailman public | file_cache | table | mailman public | headermatch | table | mailman public | listarchiver | table | mailman public | mailinglist | table | mailman public | member | table | mailman public | message | table | mailman public | onelastdigest | table | mailman public | pended | table | mailman public | pendedkeyvalue | table | mailman public | preferences | table | mailman public | template | table | mailman public | uid | table | mailman public | user | table | mailman public | workflowstate | table | mailman (24 rows)
mailman=> \q
For grins I commented out the DB url in /etc/mailman3/mailman.cfg and this is what I get:
(venv) mailman@mail:~> mailman info GNU Mailman 3.3.8 (Tom Sawyer) Python 3.11.5 (main, Aug 31 2023, 07:57:41) [GCC] config file: /etc/mailman3/mailman.cfg db url: sqlite:////opt/mailman/mm/var/data/mailman.db devmode: DISABLED REST root url: http://localhost:8001/3.1/ REST credentials: restadmin:restpass
So clearly the issue is caused by the DB url: url: postgresql://mailman:XXXXX@localhost/mailman
something between python and PostgreSQL.
(venv) mailman@mail:~> mailman info Traceback (most recent call last): File "/opt/mailman/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1077, in main with self.make_context(prog_name, args, **extra) as ctx: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 943, in make_context self.parse_args(ctx, args) File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1644, in parse_args rest = super().parse_args(ctx, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 1408, in parse_args value, args = param.handle_parse_result(ctx, opts, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result value = self.process_value(ctx, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/click/core.py", line 2362, in process_value value = self.callback(ctx, self, value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/bin/mailman.py", line 95, in initialize_config initialize(value) File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/core/initialize.py", line 229, in initialize initialize_2(propagate_logs=propagate_logs) File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/core/initialize.py", line 187, in initialize_2 config.db = getUtility(IDatabaseFactory, utility_name).create() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/database/factory.py", line 57, in create database.initialize() File "/opt/mailman/venv/lib64/python3.11/site-packages/mailman/database/base.py", line 117, in initialize self.engine = create_engine( ^^^^^^^^^^^^^^ File "<string>", line 2, in create_engine File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned return fn(*args, **kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/engine/create.py", line 601, in create_engine dbapi = dbapi_meth(**dbapi_args) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi import psycopg2 File "/opt/mailman/venv/lib64/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa SystemError: initialization of _psycopg raised unreported exception
On 10/3/23 16:42, paulrubin37@gmail.com wrote:
So clearly the issue is caused by the DB url: url: postgresql://mailman:XXXXX@localhost/mailman
something between python and PostgreSQL.
(venv) mailman@mail:~> mailman info Traceback (most recent call last):
...
File "/opt/mailman/venv/lib64/python3.11/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 690, in import_dbapi import psycopg2 File "/opt/mailman/venv/lib64/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa SystemError: initialization of _psycopg raised unreported exception
Something is broken in psycopg2-binary. Try in your venv
pip install --force-reinstall psycopg2-binary\<2.9
In your venv you need
python -c 'import psycopg2'
to succeed (i.e. return without reporting an error).
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Ok first I tried the force reinstall, that changed nothing.
(venv) mailman@mail:~> pip install --force-reinstall psycopg2-binary\<2.9 Collecting psycopg2-binary<2.9 Using cached psycopg2_binary-2.8.6-cp311-cp311-linux_x86_64.whl Installing collected packages: psycopg2-binary Attempting uninstall: psycopg2-binary Found existing installation: psycopg2-binary 2.8.6 Uninstalling psycopg2-binary-2.8.6: Successfully uninstalled psycopg2-binary-2.8.6 Successfully installed psycopg2-binary-2.8.6 (venv) mailman@mail:~> python -c 'import psycopg2' Traceback (most recent call last): File "<string>", line 1, in <module> File "/opt/mailman/venv/lib64/python3.11/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa SystemError: initialization of _psycopg raised unreported exception
Here is what is in the file that is failing, If I read this right I need to check every single sub import.
50 51 from psycopg2._psycopg import ( # noqa 52 BINARY, NUMBER, STRING, DATETIME, ROWID, 53 54 Binary, Date, Time, Timestamp, 55 DateFromTicks, TimeFromTicks, TimestampFromTicks, 56 57 Error, Warning, DataError, DatabaseError, ProgrammingError, IntegrityError, 58 InterfaceError, InternalError, NotSupportedError, OperationalError, 59 60 _connect, apilevel, threadsafety, paramstyle, 61 __version__, __libpq_version__, 62 ) 63
For grins I searched for the psycopq2 package and this may tell is something both of the 'global' pythons seem to have later versions
/opt/mailman/venv/lib/python3.11/site-packages/psycopg2 /opt/mailman/venv/lib/python3.11/site-packages/psycopg2-2.8.6.dist-info /opt/mailman.save/venv/lib/python3.8/site-packages/psycopg2 /opt/mailman.save/venv/lib/python3.8/site-packages/psycopg2_binary-2.8.6.dist-info /usr/lib64/python3.11/site-packages/psycopg2 /usr/lib64/python3.11/site-packages/psycopg2-2.9.7.dist-info /usr/local/lib64/python3.8/site-packages/psycopg2 /usr/local/lib64/python3.8/site-packages/psycopg2_binary-2.9.8.dist-info
So I went out to root root and removed the psycopq2 and binary based on the above, I also checked the python versions. This is root:
mail:/etc # python -V Python 2.7.18 mail:/etc # python3 -V Python 3.11.5 mail:/etc # pip -V pip 23.2.1 from /usr/local/lib/python3.8/site-packages/pip (python 3.8) mail:/etc # pip3 -V pip 23.2.1 from /usr/local/lib/python3.8/site-packages/pip (python 3.8) mail:/etc # pip list | grep psycopg2 << No result >>
This is mailman (venv) mailman@mail:~> python -V Python 3.11.5 (venv) mailman@mail:~> pip -V pip 23.2.1 from /opt/mailman/venv/lib64/python3.11/site-packages/pip (python 3.11) (venv) mailman@mail:~> pip list | grep psycopg2 psycopg2 2.8.6 psycopg2-binary 2.8.6
So it seems to me that wiping out the venv multiple times and recreating it produces the same result.
One last possible source of information, here are the installed postgresql packages:
S | Name | Summary | Type ---+----------------------------+-------------------------------------------------------------------------+-------- i+ | postgresql | Basic Clients and Utilities for PostgreSQL | package i+ | postgresql-contrib | Contributed Extensions and Additions to PostgreSQL | package i | postgresql-devel | PostgreSQL development header files and libraries | package i | postgresql-llvmjit | Just-in-time compilation support for PostgreSQL | package i | postgresql-llvmjit-devel | Helper package to pull all dependencies to build with llvm support | package i+ | postgresql-plperl | The PL/Tcl, PL/Perl, and PL/Python procedural languages for PostgreSQL | package i+ | postgresql-plpython | The PL/Python Procedural Languages for PostgreSQL | package i+ | postgresql-server | The Programs Needed to Create and Run a PostgreSQL Server | package i+ | postgresql-server-devel | PostgreSQL server development header files and utilities | package i | postgresql13 | Basic Clients and Utilities for PostgreSQL | package i | postgresql13-contrib | Contributed Extensions and Additions to PostgreSQL | package i | postgresql13-devel | PostgreSQL client development header files and libraries | package i | postgresql13-llvmjit | Just-in-time compilation support for PostgreSQL | package i | postgresql13-llvmjit-devel | PostgreSQL development files for extensions with LLVM support | package i | postgresql13-plperl | The PL/Tcl, PL/Perl, and PL/Python procedural languages for PostgreSQL | package i | postgresql13-plpython | The PL/Python Procedural Languages for PostgreSQL | package i | postgresql13-server | The Programs Needed to Create and Run a PostgreSQL Server | package i | postgresql13-server-devel | PostgreSQL server development header files and utilities | package i | postgresql16 | Basic Clients and Utilities for PostgreSQL | package i | postgresql16-contrib | Contributed Extensions and Additions to PostgreSQL | package i | postgresql16-llvmjit | Just-in-time compilation support for PostgreSQL | package i | postgresql16-plperl | The PL/Tcl, PL/Perl, and PL/Python procedural languages for PostgreSQL | package i | postgresql16-plpython | The PL/Python Procedural Languages for PostgreSQL | package i | postgresql16-server | The Programs Needed to Create and Run a PostgreSQL Server | package
On 10/4/23 05:09, paulrubin37@gmail.com wrote:
Ok first I tried the force reinstall, that changed nothing.
(venv) mailman@mail:~> pip install --force-reinstall psycopg2-binary\<2.9 Collecting psycopg2-binary<2.9 Using cached psycopg2_binary-2.8.6-cp311-cp311-linux_x86_64.whl
You are forcing the version to be <2.9 which is why your venvs have 2.8.6.
Note you are installing a cached version. That may be why it's broken.
You could try
pip install --no-cache-dir --force-reinstall psycopg2-binary\<2.9
or as I note below, if your Django is >=3.1
pip install --no-cache-dir -U psycopg2-binary
...
Here is what is in the file that is failing, If I read this right I need to check every single sub import.
50 51 from psycopg2._psycopg import ( # noqa 52 BINARY, NUMBER, STRING, DATETIME, ROWID, 53 54 Binary, Date, Time, Timestamp, 55 DateFromTicks, TimeFromTicks, TimestampFromTicks, 56 57 Error, Warning, DataError, DatabaseError, ProgrammingError, IntegrityError, 58 InterfaceError, InternalError, NotSupportedError, OperationalError, 59 60 _connect, apilevel, threadsafety, paramstyle, 61 __version__, __libpq_version__, 62 ) 63
This is way too much detail. This psycopg2-binary is broken. You can't successfully import psycopg2 from it. You need to get a working psycopg2 installed rather than trying to fix your broken one.
What is your Django version? The only reason for pinning psycopg2-binary<2.9 is for Django<3.1. See https://docs.mailman3.org/en/latest/install/virtualenv.html#installing-mailm.... If your Django is >=3.1 try in your venv
pip install -U psycopg2-binary
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
PROGRESS, Thank you.
mail:/etc # zypper info python311-Django Loading repository data... Reading installed packages...
Information for package python311-Django:
Repository : Main Repository (OSS) Name : python311-Django Version : 4.2.5-1.1 Arch : noarch Vendor : openSUSE Installed Size : 29.3 MiB Installed : Yes (automatically) Status : up-to-date Source package : python-Django-4.2.5-1.1.src Upstream URL : https://www.djangoproject.com Summary : A high-level Python Web framework Description : Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
(venv) mailman@mail:~> pip install --force-reinstall psycopg2-binary Collecting psycopg2-binary Obtaining dependency information for psycopg2-binary from https://files.pythonhosted.org/packages/ce/85/62825cabc6aad53104b7b6d12eb2ad... Downloading psycopg2_binary-2.9.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB) Downloading psycopg2_binary-2.9.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 6.9 MB/s eta 0:00:00 Installing collected packages: psycopg2-binary Attempting uninstall: psycopg2-binary Found existing installation: psycopg2-binary 2.8.6 Uninstalling psycopg2-binary-2.8.6: Successfully uninstalled psycopg2-binary-2.8.6 Successfully installed psycopg2-binary-2.9.9 (venv) mailman@mail:~> pip install --force-reinstall psycopg2 Collecting psycopg2 Downloading psycopg2-2.9.9.tar.gz (384 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 384.9/384.9 kB 2.2 MB/s eta 0:00:00 Preparing metadata (setup.py) ... done Building wheels for collected packages: psycopg2 Building wheel for psycopg2 (setup.py) ... done Created wheel for psycopg2: filename=psycopg2-2.9.9-cp311-cp311-linux_x86_64.whl size=505693 sha256=f0288149e9e27a5b88cc041b9385bc1f1c3a8e95958d0edf5de6d47ef3d6aadb Stored in directory: /opt/mailman/.cache/pip/wheels/ab/34/b9/78ebef1b3220b4840ee482461e738566c3c9165d2b5c914f51 Successfully built psycopg2 Installing collected packages: psycopg2 Attempting uninstall: psycopg2 Found existing installation: psycopg2 2.8.6 Uninstalling psycopg2-2.8.6: Successfully uninstalled psycopg2-2.8.6 Successfully installed psycopg2-2.9.9 (venv) mailman@mail:~> python -c 'import psycopg2' (venv) mailman@mail:~> mailman info GNU Mailman 3.3.8 (Tom Sawyer) Python 3.11.5 (main, Aug 31 2023, 07:57:41) [GCC] config file: /etc/mailman3/mailman.cfg db url: postgresql://mailman:XXXXX@localhost/mailman devmode: DISABLED REST root url: http://localhost:8001/3.1/ REST credentials: restadmin:restpass (venv) mailman@mail:~>
I went the through the rest of the install got the login screen and after logging in I get this:
Forbidden (403) CSRF verification failed. Request aborted.
Looking at the PostgreSQL db, the user table has certainly been emptied the only user is mailman. The UID, address and mailing list tables seem to contain many records. I am not sure how much to restore of the original DB? Short of doing a diff on two backup file do you have any insight?
Thanks Paul.
paulrubin37@gmail.com writes:
I went the through the rest of the install got the login screen and after logging in I get this:
Forbidden (403) CSRF verification failed. Request aborted.
This is a webserver vulnerability mitigation, and I'm seeing the same HTTP statis on mailman3.org when I go to moderate these lists. I suspect a Django bug (or barely possible, change in Django's configuration that we need to update to). But I don't know for sure what's going on there. Pretty sure it's nothing you did wrong though.
Looking at the PostgreSQL db, the user table has certainly been emptied the only user is mailman.
Is that 'user' table in the 'mailman' database? Or is that Postgres's own user table? If the latter, mailman is probably the only user authenticated by Postgres (for other users including the 'postgres' database superuser, Postgres delegates to Unix login authentication).
Since you can't get to Django for the moment, if you activate the venv and type "mailman lists", do you see all your lists? If you then type "mailman members LIST" for some list, do you see all the members?
The UID, address and mailing list tables seem to contain many records. I am not sure how much to restore of the original DB? Short of doing a diff on two backup file do you have any insight?
Why do you think that your database has been corrupted? Does Postgres complain about inconsistencies? You would hope that you lost only transactions in progress at the time of the crash, and the database is internally consistent. If it has been corrupted, you probably should restore the whole thing from the backup, but that would lose any changes committed to the database between the backup and the crash.
If any of the mailman information commands above seem to give wrong information, or worse errors, then you probably need to restore the whole thing. But I suspect the database is intact.
On 10/5/23 18:04, Stephen J. Turnbull wrote:
This is a webserver vulnerability mitigation, and I'm seeing the same HTTP statis on mailman3.org when I go to moderate these lists. I suspect a Django bug (or barely possible, change in Django's configuration that we need to update to). But I don't know for sure what's going on there. Pretty sure it's nothing you did wrong though.
I see it too now for some things on mailman3.org. It apparently just started there when I upgraded Django from 3.2 to 4.2. I'll investigate.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 10/5/23 19:28, Mark Sapiro wrote:
I see it too now for some things on mailman3.org. It apparently just started there when I upgraded Django from 3.2 to 4.2. I'll investigate.
You need to add CSRF_TRUSTED_ORIGINS to your Django settings. On lists.mailman3.org, this looks like
CSRF_TRUSTED_ORIGINS = ['https://lists.mailman3.org']
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
OK, I am up and running. It took me a couple tries I kept getting errors until I fully restarted mailmanweb and the browser
As for the user table you are correct, I was getting confused between : select * from user: and select * from public.user;
THANK YOU for all the help
Is there something I can do for you? A Donation?
On 10/6/23 05:55, paulrubin37@gmail.com wrote:
Is there something I can do for you? A Donation?
Donations to the GNU Mailman project are always appreciated. See https://wiki.list.org/x/4816967 for details.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
paulrubin37@gmail.com writes:
Note there were zero issues this time with venv or pip.
python3 -m venv venv vi .bashrc exit
You need a "source venv/bin/activate" before working in the venv, including with pip. Did you not do that, or just omit writing it in the email?
pip install wheel mailman psycopg2-binary\<2.9
participants (6)
-
Mark Sapiro -
Nelson Strother -
Odhiambo Washington -
Paul Rubin -
paulrubin37＠gmail.com
-
Stephen J. Turnbull