Some doc updates from Fedora install experience
Trying to get mailman3 setup round three. Some notes from a first timer that I hope will be of use. These are in order of experience, and unfiltered.
I'm a bit confused about why you have a release that pip should be able to install, but the advice I get is to pretend I'm developing from git. But off we go:
Following http://docs.mailman3.org/en/latest/devsetup.html:
I'm using Fedora 25. Packaged Python is 2.7.13-2, 3.6.3-6, 2-setuptools 25.1.1-1, 2-virtualenv 15.0.3-2 nodejs-less 2.7.1-1 sassc 3.4.1-1
a) fedora now uses dnf rather than yum (and has for some time). Specify dnf install rather than yum install b) sassc is packaged - just add it to the end of the dnf install string.
I'm using sendmail. Directly outbound. Inbound, a custom LMTP client.
It is possible to setup sendmail to use LMTP for local delivery - but if you do, it's the ONLY local delivery (or you're into some serious sendmail configuration hacking). Most installations use procmail and/or pipes. I could probably extract a pipe -> lmtp script from my project - but it's Perl, not Python.
The current install process assumes a lot of Python knowledge/interaction, which is a barrier to adoption.
Documentation should note that the virtualenv commands create directories of those names (I'm not a Python person, so everything is a surprise.)
To save typing, I defined: alias p0='[ -n "$_OLD_VIRTUAL_PATH" ] && deactivate;cd ~mailman3' alias p2='[ -n "$_OLD_VIRTUAL_PATH" ] && deactivate;cd ~/mailman3 && source venv-2.7/bin/activate' alias p3='[ -n "$_OLD_VIRTUAL_PATH" ] && deactivate;cd ~/mailman3 && source venv-3.5/bin/activate'
I suggest using mailman3 as the top level in your examples, since there's also a mailman directory in the path, and mailman is easily confused with mailman/mailman Also, if you have mailman2 - or your distro has provision for it - 'mailman' has other semantics. (Including, but not limited to things like SeLinux contexts...)
I got tracebacks when I got to the "mailman info" command. After some experimentation/digging, resolved with dnf install python3-dateutil
This was confusing because Python/pip seemed to think dateutil was installed - perhaps it fell back to the Python2 version?
mailman info works!
You can edit your mailman.cfg file But which one? there's one in src/mailman/config and one in var/etc/mailman.cfg. Should specify. But var looks like the one to modify, the other.
The suggestion to look at the one in src for doc isn't very helpful, there are virtually no comments. So, look at the schema.
To get started, I set the following:
[mailman] site_owner: admin@example.net [devmode] enabled: yes [mta] smtp_host: smtp.example.net smtp_port: 587 #smtp_user: #smtp_pass:
lmtp_host: 192.168.148.19 lmtp_port: 8024
### Note: LMTP needs security, preferably SSL.. Thiis is probably a bunch of config items that get fed into the Python library (e.g. use ssl, cert files (current openssl includes key w/cert, but often multiple certs - rsa + ecdsa) verify client, ca bundle, ca path, client ca.) Despite the list, these are all pass-through parameters, so should not be much work. Once you have SSL, a simple shared secret (e.g. password) can be used. Or (my preference) client verification (certificate). Maybe a proxy, but who needs another process to manage?
FWIW, my MTA runs on a separate VM from the MM instance, so assuming "localhost" is secure doesn't work. It also doesn't work on a multiuser machine.
mailman start got some things running. Telnet & LHLO wakes up LMTP!
I guess I should try to setup my first list.
More as it develops.
A number of the Python module installs generate "PEP 440" warnings, which I ignored.
There are warnings on some components - and then a failure to build HyperKity looking for redhat-hardened-cc1.
Add redhat-rpm-config to the dnf install to get that.
setup for hyperkitty - cd hyperkitty, not mailman-hyperkitty
under "setup a mail server", it says: You will also have to add some settings to your django configuration. The setup instructions are provided in django’s email documentation.
OK: What file do I put them in? Perhaps this is mailman-suite/mailman-suite_project/settings.py?
That file is a mix of stuff that an admin would want to change, and application configs for developers. Perhaps that can't be helped - but could the admin settings come first?
What's a SITE_ID ? There's no comment.
It would be nice to have a list of what settings (there are a zillion of them) need to be changed to get going. Several are authentication items (passwords, usernames, urls). A cross reference that says across the whole project, where do you need to put them would be a big help. E.g., pick your REST api url, and put it in the following files: ...
Out of time for now. Seems like sorting this out will take more work.
BTW, I've replicated this on a Fedora 23 system with Python 3.4... (VM in notebook PC for the waiting room :-)
Starting the suite - the server crashes with a deep traceback, but the root seems to be:
MissingDependency: The 'whoosh' backend requires the installation of 'Whoosh'. Please refer to the documentation.
So I ran (in the 2.7 venv) pip install Whoosh, got 2.7.4 Now I get: Request Method: GET Request URL: http://127.0.0.1:8000/ Django Version: 1.11.2 Exception Type: DisallowedHost Exception Value: Invalid HTTP_HOST header: '127.0.0.1:8000'. You may need to add u'127.0.0.1' to ALLOWED_HOSTS.
Well, let's try loclhost:8000 instead
And I get a 301 ... and eventually /postorius/lists/
Life. Everything is painfully slow, but life!
email config issue. No way to request resending confirmation. Opened an issue.
Now, where was that "email setup" for Django that I didn't find... ?
Er, the doc says the e-mail should come to the "dummy backend" via
'django.core.mail.backends.console.EmailBackend'
But while the HTTP traffic is there, no sign of the e-mail. Hmmm.
Well, find says there are 4 settings.py files:
./postorius/src/postorius/doc/settings.py
./postorius/example_project/settings.py
./mailman-suite/mailman-suite_project/settings.py
./hyperkitty/example_project/settings.py
# grep django.core.mail.backends ./postorius/src/postorius/doc/settings.py ./postorius/example_project/settings.py ./mailman-suite/mailman-suite_project/settings.py ./hyperkitty/example_project/settings.py
./mailman-suite/mailman-suite_project/settings.py:EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
./mailman-suite/mailman-suite_project/settings.py: EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
./hyperkitty/example_project/settings.py: EMAIL_BACKEND = 'django.core.mail.backends.filebased.EmailBackend'
Hmmm, must be "filebased.EmailBackend
Next, we find EMAIL_FILE_PATH = os.path.join(BASE_DIR, 'emails')
Ah - ./mailman-suite/mailman-suite_project/emails
and there, we find an e-mail with a confirmation URL!
Amazingly, it's possible to sign in.
And on that note, we end this episode of "Just reading the docs isn't quite enough"
Onwards. Seems I need to create a domain.
Two undefined terms" Web host" and "Mail host".
Web host seems to be a DNS domain name that becomes part of a pulldown.
Am I supposed to enter the names of servers (e.g. where an smtp server and httpserver live)? Or are these domain names?
lets assume smtp server and domain name.
Try to create a list. Gets more confusing. List name - that's obvious. Mail host "Choose a domain", and the prompt is "Choose a domain". But the choice is --- the smtp server? OK, another issue - 140.
Complete the list creation
HTTP Error 500: A server error occurred. Please contact yourself (I mean, 'the administrator').
Nothing unusual on the console.
Well, there's mailman-suite/mailman-suite_project/logs/mailmansuite.log - nothing there either.
./mailman/var/logs/debug.log is empty
Ah, `./mailman/var/logs/mailman.log' has
Jun 21 16:48:40 2017 (15370) command failure: /usr/sbin/postmap /root/mailman3/mailman/var/data/postfix_lmtp, 127, Key has expired
Jun 21 16:48:41 2017 (15370) command failure: /usr/sbin/postmap /root/mailman3/mailman/var/data/postfix_domains, 127, Key has expired
Jun 21 16:48:41 2017 (15370) REST request handler error:
Traceback (most recent call last):
File "/usr/lib64/python3.5/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/usr/lib/python3.5/site-packages/mailman/database/transaction.py", line 50, in wrapper
rtn = function(*args, **kws)
File "/usr/lib/python3.5/site-packages/mailman/rest/wsgiapp.py", line 214, in __call__
return super().__call__(environ, start_response)
File "/usr/lib/python3.5/site-packages/falcon/api.py", line 227, in __call__
responder(req, resp, **params)
File "/usr/lib/python3.5/site-packages/mailman/rest/lists.py", line 245, in on_post
mlist = create_list(**validator(request))
File "/usr/lib/python3.5/site-packages/mailman/app/lifecycle.py", line 100, in create_list
call_name(config.mta.incoming).create(mlist)
File "/usr/lib/python3.5/site-packages/mailman/mta/postfix.py", line 66, in create
self.regenerate()
File "/usr/lib/python3.5/site-packages/mailman/mta/postfix.py", line 103, in regenerate
raise RuntimeError(NL.join(errors))
RuntimeError: command failure: /usr/sbin/postmap /root/mailman3/mailman/var/data/postfix_lmtp, 127, Key has expired
command failure: /usr/sbin/postmap /root/mailman3/mailman/var/data/postfix_domains, 127, Key has expired
Jun 21 16:48:41 2017 (15370) 127.0.0.1 - - "POST /3.0/lists HTTP/1.1" 500 59
Well, it's true that I didn't setup postix (because I don't plan to use it). but 'key has expired' is a very strange error.
Studying the traceback, we find: /root/mailman3/mailman/var/data/postfix_lmtp contains the desired aliases
And the failure was running /usr/sbin/postmap - I guess I need to provide a replacement that captures the contents and configures my inbound mailer (instead of setting up postfix.).
It seems that this failure prevented the list setup from being committed.
/usr/sbin/postmap does not exist. "key has expired" is definitely the wrong error.
We can fix that for now...:
cat >/usr/sbin/postmap
#!/bin/bash
exit 0
chmod +x /usr/sbin/postmap
List created!
So now I can work on trying to hook up my data source...
I had to read code to discover that the web UI (Postorious) server address doesn't exist in any config file.
To bind it to something other than localhost, you put the desired ip address & port on the
python manage.py runserver command line (in mailman-suite_project.
It must be an IP address - it doesn't resolve DNS names. The port is appended to the IP address with a ':'.
So, in venv 2.7
cd mailman-suite_project && python manage.py runserver 10.0.0.11:8000 will work.
There are all sorts of issues if your network is at all public, but in my case, a multi-machine walled garden, with no local web browser, it allowed bootstrapping to continue.
The runserver command also takes --ipv6 (or -6) to bind to an IPv6 address, --nothreading and --noreload also exist (but i haven't tracked down whethere they're useful')
Hope this saves someone else a lot of research...
Excerpts from tlhackque--- via Mailman-users's message of June 24, 2017 4:21 am:
I had to read code to discover that the web UI (Postorious) server address doesn't exist in any config file.
That is True, there is no stright forward way for Mailman Core to know who is using the API and where is the Web UI working. I *think* "Web Host" is used to generate emails, but that can't always be completely accurate because there is no easy way to know what is the "root" url for Postorius and Hyperkitty. Generally, the default configurations place it at "/postorius/" and "/hyperkitty/", but there are installations where it is placed at "/mailman3/" and "/archives/".
To bind it to something other than localhost, you put the desired ip address & port on the
python manage.py runservercommand line (in mailman-suite_project.
Ok, so you should never use this in production. This command starts a "development" server which is only supposed to be used in the development environment. For any production use, in general for any python web app, we use a wsgi server which interfaces between Django and a "real" web server like Nginx or Apache. I would recommend using uwsgi. This1 and 2 might be of help.
There is some configuration over here3 that might help you get it running. You would have to change some parameters in the configuration though.
It must be an IP address - it doesn't resolve DNS names. The port is appended to the IP address with a ':'.
So, in venv 2.7
cd mailman-suite_project && python manage.py runserver 10.0.0.11:8000will work.There are all sorts of issues if your network is at all public, but in my case, a multi-machine walled garden, with no local web browser, it allowed bootstrapping to continue.
It is not supposed to be public and should only be used for development.
The runserver command also takes --ipv6 (or -6) to bind to an IPv6 address, --nothreading and --noreload also exist (but i haven't tracked down whethere they're useful')
There are a whole lot of other optins available to python manage.py command
which you can find in the documentation for Django.
Hope this saves someone else a lot of research...
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj
On 26-Jun-17 15:31, Abhilash Raj wrote:
Excerpts from tlhackque--- via Mailman-users's message of June 24, 2017 4:21 am:
I had to read code to discover that the web UI (Postorious) server address doesn't exist in any config file.
That is True, there is no stright forward way for Mailman Core to know who is using the API and where is the Web UI working. I *think* "Web Host" is used to generate emails, but that can't always be completely accurate because there is no easy way to know what is the "root" url for Postorius and Hyperkitty. Generally, the default configurations place it at "/postorius/" and "/hyperkitty/", but there are installations where it is placed at "/mailman3/" and "/archives/".
It would be helpful for API users to "register" with Mailman, so that API clients can figure out how to direct users to their web pages. There has to be a directory somewhere, and Mailman is the central character. It doesn't have to be one place. Perhaps something like "get( '/lists/config/admin_ui') and get('/lists/config/subscriber_ui") - which could return something like:
[ {"type":"gui", "name":"postorius", "uri":"https://www.example.net/mailman3/"}, {"type":"cli","name":"mailmanclient","uri":"domain://my.socket.example.net:1234"},{"type":"gui","name":"mailmanx","uri":"file:///usr/sbin/mailman-Tk"}, ...]
(Where the order indicates preference).
Speaking of which, without this knowledge, how can Mailman provide the personalized "to unsubscribe, visit your account at "...." customizations"?
I see that there are URLs for headers/footers, but we seem to have lost the ability to insert the URL for the GUI managing your list/account into personalized e-mails...
To bind it to something other than localhost, you put the desired ip address & port on the
python manage.py runservercommand line (in mailman-suite_project.Ok, so you should never use this in production. AGREEED This command starts a "development" server which is only supposed to be used in the development environment. My development environment is a VM for Mailman, and other VMs for the mail server. And if I try to run a web browser on the Mailman VM, it takes over the machine and even so is unresponsive. So I needed to make the UI visible on the network where my PC is. I do understand the issues, and there are firewalls protecting the environment.
For any production use, in general for any python web app, we use a wsgi server which interfaces between Django and a "real" web server like Nginx or Apache. I would recommend using uwsgi. This[1] and [2] might be of help.
AGREE. I know that I will have to do this eventually. But my immediate goal is to see if it's possible to get my application talking to Mailman V3 - it's a 1-user environment. So I needed an evaluation environment with the absolute minimum setup costs. If things work out, I'll do the right thing. If not, I can't afford to have spent a week or so untangling nested webservers talking to each-other with frameworks, a half-dozen passwords and config files.
This was the quickest path I could find to an 'almost out-of-the-box' evaluation and prototyping environment...and pretty much the only "step-by-step" instructions that I could find. I'd have tried "step-by-step" for production if (a) I could find it and (b) I could configure it with less than a couple of days of study :-(
I decided to document this in case it helps someone else.
I strongly agree that it's not production, and not for all users.
There is some configuration over here[3] that might help you get it running. You would have to change some parameters in the configuration though.
It must be an IP address - it doesn't resolve DNS names. The port is appended to the IP address with a ':'.
So, in venv 2.7
cd mailman-suite_project && python manage.py runserver 10.0.0.11:8000will work.There are all sorts of issues if your network is at all public, but in my case, a multi-machine walled garden, with no local web browser, it allowed bootstrapping to continue.
It is not supposed to be public and should only be used for development.
Agree, agree, agree.
The runserver command also takes --ipv6 (or -6) to bind to an IPv6 address, --nothreading and --noreload also exist (but i haven't tracked down whethere they're useful')
There are a whole lot of other optins available to
python manage.pycommand which you can find in the documentation for Django. Truth be told, I had no idea that 'runserver" had anything to do with Django. I was just following instructions to get something out of the box and onto the wire.
FWIW, 'python manage.py --help' and 'python manage.py help runserver' are semi-intructive. I still have no clue what an "auto-reloader" is, but the help does say that --noreload won't use it :-)
It's clear that a lot of work has gone into this release -- but it's also quite frustrating for people who just want to explore using it to get through the setup effort.
I've made some progress.
-- thanks, Abhilash Raj
participants (3)
-
Abhilash Raj -
tlhackque -
tlhackque＠yahoo.com