- Mailman-users - lists.mailman3.org

REST API notes
by tlhackque＠yahoo.com 29 Jun '17

29 Jun '17

As a beginner with the REST API, I found it necessary to read code and experiment to figure out how to get things done. There are a number of implementation choices that are unusual. There is some documentation in src/mailman/rest/docs. It's oriented toward Python and intermixes the mailman client & REST API, apparently trying to show equivalence. Here are some notes that may help others to avoid repeating my research. They reflect Mailman 3.1. Some reflect issues that I've opened, but I think it's worth knowing how things work today. I attempted to tie together some of pieces that are in the documentation - and note some things that aren't. I may have some things wrong, but this reflects my current understanding. (And client code.) The REST API has a hybrid interface: Requests are made with application/x-www-form-urlencoded POST, PUT, PATCH and delete http requests. Requests are also accepted as parameter strings. The responses are JSON. (This is rather surprising - one would expect JSON requests - and I hope someday they'll be accepted, as the split complicates clients. I suspect this evolved to simplify the initial Web GUI client (Postorious), but it precludes using standard JSON-in/JSON-out client libraries.) Error responses are often misidentified as content-type: application/json, but actually contain a text/plain error message. This isn't universally true; for example the 401 response actually IS JSON. A client has to guess and handle decoding exceptions. The API presents resources hierarchically, rooted at '/'. The top level resources include /users, /lists. /addresses, /domains, /system The next level is a resource id. There is a an id which is stable for the lifetime of an object, and a current name (which can be changed). For example, a list has a name like mylist(a)example.com, and a list_id of mylist.example.com. But if you change the list name, it becomes mynewname(a)example.net, while the list_id remains mylist.example.com. Lists are associated with "domains", which are the "domain part" of the list's address. That is, the part after the @. This is sometimes referred to as the "mail host", but there need not be a real host. Just something on the left side of a DNS MX record. Also, don't confuse the use of "mail host" with that of "SMTP Server", which is how Mailman sends mail. The domain is where Mailman expects to receive it. A fully qualified list name is the list name + domain - e.g. its posting address. One domain can (and usually does) support more than one list. If you delete the domain, all its lists go away too. A domain has to be created before you can create a list with an address in that domain. Resources are created with a POST to their top-level resource. To create a domain, post to /domains with mail_host => the domain, and (optionally) description => a description for the GUI. The response isn't JSON as one would expect. In fact, it's an empty application/json response with a 201 status. To create a list, one POSTs to /lists. This post takes a restricted set of parameters; in the case of a list, just its fqdn_listname, (and an optional style - which isn't well defined). The response isn't JSON as one would expect. The Location header of the response contains a URL of the new list. To configure the list, you have to follow up with a PUT or PATCH to that URL + /config This is where you can set the description, posting policy, etc. It's unrealistic to do a PUT; even if you're cloning another list, there's no programatic way to determine which attributes are writable. PATCH what you know... Members are associated with e-mail addresses - which belong to Users. You create a user by posting to /users with email => the email address, and optionally display_name => the name string. (A user can also be implicitly created by subscribing an e-mail address, but that gets confusing.) The e-mail address is the primary email address for the user. More later. Again, you get a Location header in the response, which you can use to PATCH /preferences to set delivery_status, etc. .These preferences are part of a hierarchy - many have a system default, a list default, the user default, and a subscription to that list value. You can find the User associated with a list by a GET of /addresses/address(a)example.net. This GET returns two URLs: user => the user owning this address, and self_link => the address object. Note also that there are both an email and original_email attribute. The latter preserves case. The former is used internally by Mailman as a resource key, etc. (Though exactly what happens if John(a)example.net and john(a)example.net both subscribe is unclear.) Once a list is configured, you can add members. This requires the list_id - which you don't (officially) have. So you do a GET on the list resource, to get the list_id. Then you can subscribe a member with list_id => (the list id), subscriber => email address. Optionally you can pre_verify/confirm/approve the member and/or add a display_name.. POST to /members. Again, you get a Location header back. You can't specify everything you might like. as a creation attribute. So you may have to PATCH the member to, for example, set the moderation_status. You may also need to PATCH the member /preferences if you want to set list-level delivery status, etc. Consider adding at least an owner when creating a list. One challenge is that almost everything requires multiple REST operations to set up. But REST (by definition) is stateless. So the best you can do is order operations & hope. The mailman client examples refer to transactions (e.g in users.rst, there is 'transaction.commit(); - but REST can't hold state. It does appear that the server uses DB transactions to ensure that any given REST operation is ACID, but the composite operations (e.g. create a list and set it's config) can not be Atomic. This is an architectural flaw in the API. Symbolic names (which are required) for attributes are in the src/mailman/interfaces/(class).py. What you can get/set comes from src/mailman/rest/(class).py.

3 2

Some doc updates from Fedora install experience
by tlhackque＠yahoo.com 26 Jun '17

26 Jun '17

Trying to get mailman3 setup round three. Some notes from a first timer that I hope will be of use. These are in order of experience, and unfiltered. I'm a bit confused about why you have a release that pip should be able to install, but the advice I get is to pretend I'm developing from git. But off we go: Following http://docs.mailman3.org/en/latest/devsetup.html: I'm using Fedora 25. Packaged Python is 2.7.13-2, 3.6.3-6, 2-setuptools 25.1.1-1, 2-virtualenv 15.0.3-2 nodejs-less 2.7.1-1 sassc 3.4.1-1 a) fedora now uses dnf rather than yum (and has for some time). Specify dnf install rather than yum install b) sassc is packaged - just add it to the end of the dnf install string. I'm using sendmail. Directly outbound. Inbound, a custom LMTP client. It is possible to setup sendmail to use LMTP for local delivery - but if you do, it's the ONLY local delivery (or you're into some serious sendmail configuration hacking). Most installations use procmail and/or pipes. I could probably extract a pipe -> lmtp script from my project - but it's Perl, not Python. The current install process assumes a lot of Python knowledge/interaction, which is a barrier to adoption. Documentation should note that the virtualenv commands create directories of those names (I'm not a Python person, so everything is a surprise.) To save typing, I defined: alias p0='[ -n "$_OLD_VIRTUAL_PATH" ] && deactivate;cd ~mailman3' alias p2='[ -n "$_OLD_VIRTUAL_PATH" ] && deactivate;cd ~/mailman3 && source venv-2.7/bin/activate' alias p3='[ -n "$_OLD_VIRTUAL_PATH" ] && deactivate;cd ~/mailman3 && source venv-3.5/bin/activate' I suggest using mailman3 as the top level in your examples, since there's also a mailman directory in the path, and mailman is easily confused with mailman/mailman Also, if you have mailman2 - or your distro has provision for it - 'mailman' has other semantics. (Including, but not limited to things like SeLinux contexts...) I got tracebacks when I got to the "mailman info" command. After some experimentation/digging, resolved with dnf install python3-dateutil This was confusing because Python/pip seemed to think dateutil was installed - perhaps it fell back to the Python2 version? mailman info works! >> You can edit your mailman.cfg file But which one? there's one in src/mailman/config and one in var/etc/mailman.cfg. Should specify. But var looks like the one to modify, the other. The suggestion to look at the one in src for doc isn't very helpful, there are virtually no comments. So, look at the schema. To get started, I set the following: [mailman] site_owner: admin(a)example.net [devmode] enabled: yes [mta] smtp_host: smtp.example.net smtp_port: 587 #smtp_user: #smtp_pass: lmtp_host: 192.168.148.19 lmtp_port: 8024 ### Note: LMTP needs security, preferably SSL.. Thiis is probably a bunch of config items that get fed into the Python library (e.g. use ssl, cert files (current openssl includes key w/cert, but often multiple certs - rsa + ecdsa) verify client, ca bundle, ca path, client ca.) Despite the list, these are all pass-through parameters, so should not be much work. Once you have SSL, a simple shared secret (e.g. password) can be used. Or (my preference) client verification (certificate). Maybe a proxy, but who needs another process to manage? FWIW, my MTA runs on a separate VM from the MM instance, so assuming "localhost" is secure doesn't work. It also doesn't work on a multiuser machine. mailman start got some things running. Telnet & LHLO wakes up LMTP! I guess I should try to setup my first list. More as it develops.

3 6

Mailman 3 Container Images v0.1
by Abhilash Raj 26 Jun '17

26 Jun '17

Hi Everyone, I have been working on containerizing Mailman 3 so that it becomes easier to test and try Mailman 3. There are two images in total: * maxking/mailman-core : This consists of Mailman Core and includes the plugin to enable the Hyperkitty Archiver. * maxking/mailman-web : This consists of a full Django installation and run the Mailman's Web UI components. It has Mailman Client, Postorius & Hyperkitty. The images are built from the Dockerfiles hosted on Github[3] in case you want to look at it. You can find most of the documentation about these images at https://asynchronous.in/docker-mailman/. The version of the internal components included in each image is available here[7]. The images are versioned and the process is documented[4], please have a look before you use them. Also, the images are signed and I encourage you to verify them before using. The process of how-to verify is also documented. Docker Compose is a tool written in Python makes is very easy to deploy connected-images and take the pain out of writing long long commands for spinning of containers. There is a sample `docker-compose.yml` file in the Github repository mentioned above which is kind-of tested to be able to spin off these images in vanilla condition (without any customization/configuration). You can report issues, comments or concerns in the Github repo[6]. [1]: https://hub.docker.com/r/maxking/mailman-core/ [2]: https://hub.docker.com/r/maxking/mailman-web/ [3]: https://github.com/maxking/docker-mailman [4]: https://asynchronous.in/docker-mailman/#release [5]: http://semver.org/ [6]: https://github.com/maxking/docker-mailman/issues [7]: https://github.com/maxking/docker-mailman/wiki -- thanks, Abhilash Raj

1 0

API Question: map list -> UI URL?
by tlhackque＠yahoo.com 24 Jun '17

24 Jun '17

Using the REST API, how do I get from a list / list configuration to the URL of the management GUI? I've looked through the data returned by the API and even the SQL schemas; I don't see the path. Am I missing something, or do I need to log an issue for this? I'm not particularly interested in the subscriber pages, but it ought to be possible to get there too... Thanks.

1 0

Topics? First bug...
by tlhackque 21 Jun '17

21 Jun '17

Hi, I'd like to deploy a new project on Mailman 3, which I prototyped on 2.1). Unfortunately, it's organized as one mailing list with ~ 120 topics. (The project follows another source, and it's much easier to spin up (or down) a new topic than to instantiate & manage a new list.) I see that the latest Mailman 3 release (Congrats!) doesn't support topics yet. Where is that on your todo list? First bug: Looking at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/. If I set "results per page" to a large number & switch months, results per page reverts to the default (10). It should stick with whatever I select... It also would be nice to have an account preference for this, so it doesn't have to be set on each visit.

2 2

Re: Http/Https
by Abhilash Raj 15 Jun '17

15 Jun '17

Excerpts from Simon HANNA's message of June 14, 2017 1:06 pm: > You are supposed to force https in the webserver, serving postorius and hyperkitty. Especially in postorius almost all pages contain sensitive information. > > On June 14, 2017 10:05:59 PM GMT+02:00, Bill <bill3(a)uniserve.com> wrote: >>Hi folks, >> >>We're setting up Mailman 3.1 and so far installation has been good and >>we're now in the testing phase. >> >>We've run into a snag with the account signup function on Postorius. >> >>When Postorius sends an email to the user to confirm their address, it >>displays a link using the https protocol. This doesn't work. But, if we >> >>manually change the protocol to http it will work, confirm the address >>and take us back to Postorius. >> >>Is this a known bug or have we committed a configuration error? I agree to Simon's suggestion that you should use HTTPS for all communications for Postorius. However, if you really want to, it is possible to change the behavior using a configuration parameter in django-allauth which is what controls the user signup and email verification. ACCOUNT_DEFAULT_HTTP_PROTOCOL (=”http”) If you set the above to "http" in your django's settings.py, it should then generate urls with http. [1]: https://django-allauth.readthedocs.io/en/latest/configuration.html Message-ID: <draft-878tktptc9.fsf(a)gmail.com> Date: Thu, 15 Jun 2017 01:07:18 -0700 X-Notmuch-Emacs-Draft: True -- thanks, Abhilash Raj

1 0

Http/Https
by Bill 14 Jun '17

14 Jun '17

Hi folks, We're setting up Mailman 3.1 and so far installation has been good and we're now in the testing phase. We've run into a snag with the account signup function on Postorius. When Postorius sends an email to the user to confirm their address, it displays a link using the https protocol. This doesn't work. But, if we manually change the protocol to http it will work, confirm the address and take us back to Postorius. Is this a known bug or have we committed a configuration error? All assistance appreciated. Bill -- Sent using Icedove on Debian GNU/Linux.

2 1

Re: "Private archives" aren't private
by Simon Liebold 06 Jun '17

06 Jun '17

Am 06.06.2017 um 22:25 schrieb Abhilash Raj: > Excerpts from Simon Liebold via Mailman-users's message of June 6, > 2017 1:20 pm: >> >> AttributeError: type object 'BaseCommand' has no attribute >> 'option_list' > > It was reported recently on the bug tracker, you can track the > progress here[1]. > > > [1]: https://gitlab.com/mailman/hyperkitty/issues/135#note_31529582 Thanks Abhilash for the pointer. Simon

1 0

(Email|Thread) matching query does not exist ?
by Simon Liebold 06 Jun '17

06 Jun '17

I am seeing these errors: mailman-suite_project$ python manage.py qcluster 19:27:52 [Q] INFO Q Cluster-11541 starting. 19:27:52 [Q] INFO Process-1:1 ready for work at 11546 19:27:52 [Q] INFO Process-1:2 ready for work at 11547 19:27:52 [Q] INFO Process-1 guarding cluster at 11545 19:27:52 [Q] INFO Process-1:4 pushing tasks at 11549 19:27:52 [Q] INFO Q Cluster-11541 running. 19:27:52 [Q] INFO Process-1:3 monitoring at 11548 19:27:52 [Q] INFO Process-1:1 processing [rebuild_email_cache_votes] 19:27:52 [Q] INFO Process-1:2 processing [rebuild_thread_cache_votes] 19:27:52 [Q] INFO Process-1:1 processing [rebuild_email_cache_votes] 19:27:52 [Q] ERROR Failed [rebuild_email_cache_votes] - Email matching query does not exist. 19:27:52 [Q] ERROR Failed [rebuild_email_cache_votes] - Email matching query does not exist. 19:27:52 [Q] ERROR Failed [rebuild_thread_cache_votes] - Thread matching query does not exist. Should I need to worry? I recently deleted some test emails from the archive. Don't know if this has anything to do with it. Simon

1 0

"Delete this thread" -> 'HTTP-500 -> NoneType' object has no attribute 'all'
by Simon Liebold 30 May '17

30 May '17

Is deleting threads from the archive (via Hyperkitty) currently supported? I am getting the error indicated above if I click on "Delete this thread". Is there another way to delete whole threads or individual posts? Simon

3 7