Mailman installation results in KeyError: 'env_py'
Hi,
I'm trying to install Mailman 3 on Debian 9. However, it doesn't let me do.
$ virtualenv -p python3 env
[...]
$ . ./env/bin/activate
$ pip install mailman
[...]
Successfully installed [...]mailman-3.1.0[...]
$ mailman info
Traceback (most recent call last):
File "/home/mailman/mailman-test/env/bin/mailman", line 11, in <module>
sys.exit(main())
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/mailman/bin/mailman.py", line 97, in main
initialize(config_path)
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/mailman/core/initialize.py", line 185, in initialize
initialize_2(propagate_logs=propagate_logs)
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/mailman/core/initialize.py", line 156, in initialize_2
config.db = getUtility(IDatabaseFactory, utility_name).create()
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/mailman/database/factory.py", line 55, in create
SchemaManager(database).setup_database()
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/mailman/database/factory.py", line 96, in setup_database
alembic.command.stamp(alembic_cfg, 'head')
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/alembic/command.py", line 504, in stamp
script.run_env()
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/alembic/script/base.py", line 425, in run_env
util.load_python_file(self.dir, 'env.py')
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
module = load_module_py(module_id, path)
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/alembic/util/compat.py", line 83, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 673, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/mailman/database/alembic/env.py", line 38, in <module>
@public
File "/home/mailman/mailman-test/env/lib/python3.5/site-packages/public/public.py", line 15, in public
else sys.modules[thing.__module__].__dict__)
KeyError: 'env_py'
I'm not a Python developer, I usually use Ruby for anything web-related, thus I can't make anything of this. Since the error message conveys exactly no information that I could use to find the problem and Google provides no mailman-specific information on it, I don't know what to do now. Any hint is appreciated.
Python version, if of any use:
$ python --version
Python 3.5.3
Greetings Marvin
-- Blog: https://www.guelkerdev.de PGP/GPG ID: F1D8799FBCC8BC4F
On 10/22/2017 03:40 AM, Marvin Gülker wrote:
Hi,
I'm trying to install Mailman 3 on Debian 9. However, it doesn't let me do. ... KeyError: 'env_py'
This is the issue reported at <https://gitlab.com/mailman/mailman/issues/423>. It is caused by a change in Alembic 0.9.6. If you downgrade Alembic to 0.9.5, it will work, or you can install the fix at <https://gitlab.com/mailman/mailman/commit/24f2439aeabb5919d387d84aa9f66b5d79...>
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 22. October 2017 um 07:49 Uhr -0700 schrieb Mark Sapiro <mark@msapiro.net>:
or you can install the fix at
Thanks. Applying this patch makes it work. Now I've got a different problem, though. I can't get it to connect to Postfix for sending emails it appears. After I created a superuser account using
$ python manage.py createsuperuser
as described on <http://postorius.readthedocs.io/en/latest/setup.html>, I tried to login using this credentials into Postorius. It then reported it sent a confirmation email to me (for an account I created on commandline?!), but Postfix' logs reveal that it didn't; there's not even a sign of an attempted delivery in /var/log/mail.log. I've configured the Postfix configuration in mailman.cfg like this:
[mta]
incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: localhost
smtp_port: 25
configuration: python:mailman.config.postfix
as was outlined on <http://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html>.
mailman-suite's settings.py contains:
# Change this when you have a real email backend
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'localhost'
EMAIL_PORT = 25
which was described on <http://docs.list.org/en/latest/config-web.html>.
For completeness, the following is in Postfix' main.cf, but is not the cause of the problem as it affects the incoming direction, whereas I refer to the outgoing one:
transport_maps = hash:/home/mailman/var/data/postfix_lmtp
local_recipient_maps = hash:/home/mailman/var/data/postfix_lmtp
I failed to locate any kind of logging information about the failed email, so I don't even know where I should start looking for the problem. var/logs/smtp.log is empty in the mailman directory, and likewise is logs/mailmansuite.log in the mailman-suite_project directory.
That being said, the enormous number of configuration files to consider and dozens of cross-references to other pages in the docs make the installation guide hard to follow; it also lacks logical structure in my opinion, when it refers to parts of the configuration which are later explained. For someone like me who never programmed a Django application (Ruby dev) this is all pretty cryptic. Especially, that some parts of information are on <http://docs.list.org/en/latest/index.html> whereas other parts are scattered over readthedocs.io makes it very complicated.
Greetings Marvin
-- Blog: https://www.guelkerdev.de PGP/GPG ID: F1D8799FBCC8BC4F
Thanks. Applying this patch makes it work. Now I've got a different problem, though. I can't get it to connect to Postfix for sending emails it appears. After I created a superuser account using
$ python manage.py createsuperuser
as described on <http://postorius.readthedocs.io/en/latest/setup.html>, I tried to login using this credentials into Postorius. It then reported it sent a confirmation email to me (for an account I created on commandline?!), but Postfix' logs reveal that it didn't; there's not even a sign of an attempted delivery in /var/log/mail.log. I've configured the Postfix configuration in mailman.cfg like this:
mailman-suite's settings.py contains:
# Change this when you have a real email backend EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_PORT = 25
There is a second occurrence of EMAIL_BACKEND in the settings file. Try commenting it, or setting DEBUG to True IIRC I have created an issue for that on gitlab.
The email you are referring to, is sent by Postorius aka Django and not Mailman. So you won't find it in the logs of Mailman. The email is being sent by the library that Postorius uses for authentication and user management. I'll see if there is a switch to autoconfirm such accounts.
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Sun, Oct 22, 2017, at 11:23 AM, Simon Hanna wrote:
Thanks. Applying this patch makes it work. Now I've got a different problem, though. I can't get it to connect to Postfix for sending emails it appears. After I created a superuser account using
$ python manage.py createsuperuser
as described on <http://postorius.readthedocs.io/en/latest/setup.html>, I tried to login using this credentials into Postorius. It then reported it sent a confirmation email to me (for an account I created on commandline?!), but Postfix' logs reveal that it didn't; there's not even a sign of an attempted delivery in /var/log/mail.log. I've configured the Postfix configuration in mailman.cfg like this:
mailman-suite's settings.py contains:
# Change this when you have a real email backend EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_PORT = 25
There is a second occurrence of EMAIL_BACKEND in the settings file. Try commenting it, or setting DEBUG to True IIRC I have created an issue for that on gitlab.
Setting DEBUG=True will use the 2nd EMAIL_BACKEND
defined in settings
file 1, which I think is the problem.
Try setting DEBUG=False to use 'django.core.mail.backends.emtp.EmailBackend' backend, also, please feel free to remove the 2nd definition of EMAIL_BACKEND 1 if you have Postfix already setup to send emails. That extra option is left there so that people can debug their Django installation even without having to setup Postfix.
-- Abhilash Raj maxking@asynchronous.in
On Sun, Oct 22, 2017, at 09:36 AM, Marvin Gülker wrote:
Am 22. October 2017 um 07:49 Uhr -0700 schrieb Mark Sapiro <mark@msapiro.net>:
or you can install the fix at
Thanks. Applying this patch makes it work. Now I've got a different problem, though. I can't get it to connect to Postfix for sending emails it appears. After I created a superuser account using
$ python manage.py createsuperuser
as described on <http://postorius.readthedocs.io/en/latest/setup.html>, I tried to login using this credentials into Postorius. It then reported it sent a confirmation email to me (for an account I created on commandline?!), but Postfix' logs reveal that it didn't; there's not even a sign of an attempted delivery in /var/log/mail.log. I've configured the Postfix configuration in mailman.cfg like this:
[mta] incoming: mailman.mta.postfix.LMTP outgoing: mailman.mta.deliver.deliver lmtp_host: 127.0.0.1 lmtp_port: 8024 smtp_host: localhost smtp_port: 25 configuration: python:mailman.config.postfix
as was outlined on <http://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html>.
mailman-suite's settings.py contains:
# Change this when you have a real email backend EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_PORT = 25
which was described on <http://docs.list.org/en/latest/config-web.html>.
For completeness, the following is in Postfix' main.cf, but is not the cause of the problem as it affects the incoming direction, whereas I refer to the outgoing one:
transport_maps = hash:/home/mailman/var/data/postfix_lmtp local_recipient_maps = hash:/home/mailman/var/data/postfix_lmtp
I failed to locate any kind of logging information about the failed email, so I don't even know where I should start looking for the problem. var/logs/smtp.log is empty in the mailman directory, and likewise is logs/mailmansuite.log in the mailman-suite_project directory.
As Simon mentioned, try using DEBUG=False, if you don't already have
that. With DEBUG=True, you should see emails printed in a emails
directory under mailman-suite_project
directory.
This is a confirmation email to verify the email addresses, which happens even if you create an account from command line.
That being said, the enormous number of configuration files to consider and dozens of cross-references to other pages in the docs make the installation guide hard to follow; it also lacks logical structure in my opinion, when it refers to parts of the configuration which are later explained. For someone like me who never programmed a Django application (Ruby dev) this is all pretty cryptic. Especially, that some parts of information are on <http://docs.list.org/en/latest/index.html> whereas other parts are scattered over readthedocs.io makes it very complicated.
There are only two primary configuration files (there are a few more for finer configurations ;-):
mailman.cfg: This is the configuration for the Core engine and can be located any of the places mentioned in 1 or pointed to by environment var MAILMAN_CONFIG_FILE.
settings.py: This is the configuration file for Django under
mailman-suite_project
and controls the frontend (Postorius & Hyperkitty). Django is a huge framework with literally 1000s of options. It would be impossible to copy that information from Django's documentation and keep it updated in our docs. Hence, we have links to appropriate places when needed. Same goes out for the rest of the libraries that we use.
All the settings mentioned in the documentation go to either of these files, except, when configuring web server and mail server. Please open an issue2 if it is not clear where a particular configuration snippet goes and we will fix it.
Hope that helps!
-- Abhilash Raj maxking@asynchronous.in
Hi,
Am 22. October 2017 um 11:47 Uhr -0700 schrieb Abhilash Raj <maxking@asynchronous.in>:
As Simon mentioned, try using DEBUG=False, if you don't already have that. With DEBUG=True, you should see emails printed in a
emails
directory undermailman-suite_project
directory.
Bingo! Man, I was not seeing the forest for the trees. I should have peeked inside that emails folder. And yes, changing the DEBUG flag resolves the problem and emails now go out as they should. My bad! Thank you!
That being said, the enormous number of configuration files to consider and dozens of cross-references to other pages in the docs make the installation guide hard to follow; it also lacks logical structure in my opinion, when it refers to parts of the configuration which are later explained. For someone like me who never programmed a Django application (Ruby dev) this is all pretty cryptic. Especially, that some parts of information are on <http://docs.list.org/en/latest/index.html> whereas other parts are scattered over readthedocs.io makes it very complicated.
There are only two primary configuration files (there are a few more for finer configurations ;-):
Mh, mailman.cfg, settings.py, and hyperkitty.py are three. But I agree, I exaggerated.
All the settings mentioned in the documentation go to either of these files, except, when configuring web server and mail server. Please open an issue[2] if it is not clear where a particular configuration snippet goes and we will fix it.
I'll open an issue later; for now off the top of my head so I won't forget I came over these problems with the docs:
- The linked installation guide for Sass (<http://sass-lang.com/install>) does not include any C/C++ implementation you speak of.
- The mailman-suite's settings.py includes a handler for Less, which is not mentioned in any documentation.
- It's officially suggested to use the "fhs" filesystem path layout. It's not documented that you need to create the exact directories and change permissions accordingly; even worse, if one uses this layout, Mailman tries to write into /sbin, which only the package manager is allowed to do. Thus, don't recommend that layout in <http://docs.list.org/en/latest/config-core.html>.
- This page: <http://docs.list.org/en/latest/config-core.html> does not mention that the "postfix_lmtp" and "postfix_domains" files are *generated* by Mailman. I spend quite some time looking for these files before I found in the Mailman Core docs that they're generated and it was not an error in my installation.
- This page: <http://docs.list.org/en/latest/config-core.html> configures the LMTP server and the REST API server for the same TCP port (8024). This clashes.
- This page: <http://docs.list.org/en/latest/config-web.html> should mention the thing with the DEBUG variable we've successfully worked through in this thread.
- There's no mention of how to create the initial superuser on <http://docs.list.org/en/latest/>, it is hidden down in <https://postorius.readthedocs.io/en/latest/setup.html>, where however there's no mention of the mailman-suite, but some kind of "example_project". It was unclear to me whether this now applied to me or not.
- <http://docs.list.org/en/latest> contains quite some pip install commands (duplicated from the postorius/hyperkitty/core docs), but it's missing "$ pip3 install mailman-hyperkitty", which is only mentioned in <http://hyperkitty.readthedocs.io/en/latest/install.html> in the text body (but not as a command block).
- This page: <http://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html> uses "configuration: python:mailman.config.postfix" in the introductory text without explanation, but down in the Postfix-specific block the directive is missing with no explanation. I opted to include it, but I don't know whether that's now correct or not.
The overall problem I had with the docs is that they can't be followed top-down. One has to refer to the subproject docs if something is unclear, then searching in them until one finds what one was looking for (even if one doesn't know what one is looking for, as with the superuser creation command). I totally appreciate the detailed subproject-docs approach, but you should follow it consequently then. That is, do not duplicate some information from the subproject docs in the mailman-suite docs. It creates the impression as if the mailman-suite docs are complete, which they aren't. Instead, the mailman-suite docs should use explicit references, for example:
- Do this...
- Now follow the procedure outlined in [Postorius Installation link]
- Do something else...
- Follow the procudedure in [Hyperkitty installation link]
That way, the Mailman installation guide would be less of a puzzle. It however might just be me. I apologise if I sound silly.
Greetings Marvin
-- Blog: https://www.guelkerdev.de PGP/GPG ID: F1D8799FBCC8BC4F
On Sun, Oct 22, 2017, at 01:28 PM, Marvin Gülker wrote:
Hi,
Am 22. October 2017 um 11:47 Uhr -0700 schrieb Abhilash Raj <maxking@asynchronous.in>:
As Simon mentioned, try using DEBUG=False, if you don't already have that. With DEBUG=True, you should see emails printed in a
emails
directory undermailman-suite_project
directory.Bingo! Man, I was not seeing the forest for the trees. I should have peeked inside that emails folder. And yes, changing the DEBUG flag resolves the problem and emails now go out as they should. My bad! Thank you!
That being said, the enormous number of configuration files to consider and dozens of cross-references to other pages in the docs make the installation guide hard to follow; it also lacks logical structure in my opinion, when it refers to parts of the configuration which are later explained. For someone like me who never programmed a Django application (Ruby dev) this is all pretty cryptic. Especially, that some parts of information are on <http://docs.list.org/en/latest/index.html> whereas other parts are scattered over readthedocs.io makes it very complicated.
There are only two primary configuration files (there are a few more for finer configurations ;-):
Mh, mailman.cfg, settings.py, and hyperkitty.py are three. But I agree, I exaggerated.
All the settings mentioned in the documentation go to either of these files, except, when configuring web server and mail server. Please open an issue[2] if it is not clear where a particular configuration snippet goes and we will fix it.
I'll open an issue later; for now off the top of my head so I won't forget I came over these problems with the docs:
- The linked installation guide for Sass (<http://sass-lang.com/install>) does not include any C/C++ implementation you speak of.
Fixed, http://sass-lang.com/libsass is the correct URL, not sure why is
it not mentioned under /install
.
- The mailman-suite's settings.py includes a handler for Less, which is not mentioned in any documentation.
I don't think we use less
anymore, I will confirm this and remove it
if needed. It hasn't caused a problem yet, so I am assuming it is never
used.
https://gitlab.com/mailman/mailman-suite/issues/7
- It's officially suggested to use the "fhs" filesystem path layout. It's not documented that you need to create the exact directories and change permissions accordingly; even worse, if one uses this layout, Mailman tries to write into /sbin, which only the package manager is allowed to do. Thus, don't recommend that layout in <http://docs.list.org/en/latest/config-core.html>.
I have been thinking which layout is best and should be used. FHS makes
most sense, but it is true that pip
can't write to /sbin
and hence
it doesn't work if you aren't using a package manager.
I have changed the docs to recommend master
layout which uses the
whichever directory the mailman
command is stored in as bin_dir
.
- This page: <http://docs.list.org/en/latest/config-core.html> does not mention that the "postfix_lmtp" and "postfix_domains" files are *generated* by Mailman. I spend quite some time looking for these files before I found in the Mailman Core docs that they're generated and it was not an error in my installation.
Added a note about it in the docs under "Configuring MTA" section.
- This page: <http://docs.list.org/en/latest/config-core.html> configures the LMTP server and the REST API server for the same TCP port (8024). This clashes.
Fixed, thanks!
- This page: <http://docs.list.org/en/latest/config-web.html> should mention the thing with the DEBUG variable we've successfully worked through in this thread.
Added a note about it.
- There's no mention of how to create the initial superuser on <http://docs.list.org/en/latest/>, it is hidden down in <https://postorius.readthedocs.io/en/latest/setup.html>, where however there's no mention of the mailman-suite, but some kind of "example_project". It was unclear to me whether this now applied to me or not.
Added a section about it under http://docs.mailman3.org/en/latest/config-web.html
- <http://docs.list.org/en/latest> contains quite some pip install commands (duplicated from the postorius/hyperkitty/core docs), but it's missing "$ pip3 install mailman-hyperkitty", which is only mentioned in <http://hyperkitty.readthedocs.io/en/latest/install.html> in the text body (but not as a command block).
Added to the installation guide.
- This page: <http://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html> uses "configuration: python:mailman.config.postfix" in the introductory text without explanation, but down in the Postfix-specific block the directive is missing with no explanation. I opted to include it, but I don't know whether that's now correct or not.
You only need to specify settings that you want to override from default. "configuration" for Postfix used to be completely blank until recently, when we added support for regex maps along with hash maps in Postfix. The default is set to hash maps and you don't need to specify the configuration if you using it, if you need to use regex maps, you create a new configuration file and point to it under "configuration".
I will see how to make that more clear in there.
The overall problem I had with the docs is that they can't be followed top-down. One has to refer to the subproject docs if something is unclear, then searching in them until one finds what one was looking for (even if one doesn't know what one is looking for, as with the superuser creation command). I totally appreciate the detailed subproject-docs approach, but you should follow it consequently then. That is, do not duplicate some information from the subproject docs in the mailman-suite docs. It creates the impression as if the mailman-suite docs are complete, which they aren't. Instead, the mailman-suite docs should use explicit references, for example:
- Do this...
- Now follow the procedure outlined in [Postorius Installation link]
- Do something else...
- Follow the procudedure in [Hyperkitty installation link]
That way, the Mailman installation guide would be less of a puzzle. It however might just be me. I apologise if I sound silly.
It wasn't silly at all, in-fact you comments were very helpful to improve the documentation. We would like to not make assumptions about pre-requisite knowledge to use Mailman's documentation and it is hard for us developers to do that without feedback.
Thanks a lot to you!
-- Abhilash Raj maxking@asynchronous.in
Hi,
thank you for your quick reaction on my feedback. That was really fast!
I've got my mailman instance now up and running mostly, just some non-mailman-specific parts need to be done (DNS, TLS). Two last issues I found in the docs:
- This page: <http://docs.list.org/en/latest/prodsetup.html#installing-web-ui> says to run the migrations via "migrate.py". To the contrary, this page: <http://hyperkitty.readthedocs.io/en/latest/install.html#setting-up-the-datab...> advertises to run them via "django-admin". I assume that both commands are equivalent (and opted for the first one), but you might want to align those.
- It was unclear to me what value I am supposed to set for the
base_url' option in hyperkitty.cfg. I run the Mailman-Suite on a (Hiawatha, but that doesn't matter) VirtualHost so
localhost` obviously doesn't work. By trial&error I found that I need to set this option to the public Hyperkitty path (including with HTTPS), which is logical, but the comment in the provided config file at <https://gitlab.com/mailman/mailman-hyperkitty/blob/master/mailman-hyperkitty...> (linked to from <http://hyperkitty.readthedocs.io/en/latest/install.html#connecting-to-mailma...>) made me think that's wrong because it was a public URL, whereas the comment recommends it to be private. Are Hyperkitty archives not meant to be publically used? Or what's the intention behind this comment?
Thank you everyone! Marvin
-- Blog: https://www.guelkerdev.de PGP/GPG ID: F1D8799FBCC8BC4F
participants (4)
-
Abhilash Raj
-
Mark Sapiro
-
Marvin Gülker
-
Simon Hanna