Search results for "Postorius" - Mailman-users

Re: Which folder should I share between pods with kubernetees persistentVolumeClaim ?

by Stephen J. Turnbull

antoine.depoisier--- via Mailman-users writes: > First, my nodes are all an instance of mailman-core only. > I have an existing MTA that handle all email, and sent them to a > proxy, that dispatch all email between the existing pods. Then, > email are transferred to the MTA and MTA send them. So you are distributing mailman core across multiple nodes? And "proxy" = "load balancer" (such as HAProxy)? That is probably subject to a number of problems, unless you do things that I'm not sure how to do. For example, when a user account is disabled by bounces, every so often Mailman will send a "hey, are you alive again?" message to the user. If enough of those bounce, the user gets unsubscribed. The problem is that that status is recorded in the PostgreSQL database that all Mailman instances have access to, and I think they'll probably all send those messages. At best there's a race condition where more than one might send the message. Under some conditions moderators may get daily messages about bounces. I suspect they would get one from each Mailman instance. I think digests will be broken unless you share the lists subfolder, because each Mailman instance will accumulate only the messages it processes, so chances are good on an active list that digest subscribers will get multiple partial digests when they should get only one. As I'll describe below, Mailman tends to spawn a lot of processes, many of which don't have much work to do. Now you're dividing very little work across multiple nodes, which seems very wasteful. You haven't said where your MTA lives. If it's Postfix, it needs to share the $data_dir folder with a Mailman instance that is responsible for list creation. Every time you restart Mailman it recreates the files in that folder, so if it's shared among Mailman instances there will be delays due to file locking. So unless you have an insane amount of list traffic to process (say, a G7 national government ;-), I wonder if the multi-instance approach is necessary. Mailman is not implemented to be operated that way -- you're asking for trouble. The design is such that I imagine it can be done with careful tuning, but current default configuration didn't consider such a use case. You don't need to answer, you know your use case and I don't, but you may save yourself a lot of trouble by just running with a little more resources on a single node. Mailman uses a lot of memory just to get started (about 2GB unless you're really aggressive and do unsupported tuning of which runners are started and what modules are loaded), but then it easily scales without increasing resources except CPU to some extent. For example I've worked on a system that processes hundreds of thousands of incoming posts a day on a single Linode (2vCPU, 16GB) running core, Postorius, HyperKitty, Xapian, and nginx (PostgreSQL got its own VM for some reason). CPU usage on that system never gets above 25%, active memory usage generally 20-25% and there's usually a substantial amount free despite Linux's strategy of aggressively caching files, load average normally around 2.5 and never more than 5. The only tuning we did there was to bump the out queue's slices to 8 and in queue to 2, but all running on that same Linode (which pushes Mailman's memory usage to over 2GB). Running "ls -R queue" gives all empty subfolders about 2/3 of the time. > For the folder queue, I'm not sure if I should share this folder > between all pods, because one instance of mailman core mean one > worker if I'm right, No, one instance of Mailman core means a minimum of 15 processes (one master and 14 runners) in the default configuration. About half of those have nothing to do most of the time. You can probably whittle that 15 down to 11 with some care at a cost of a certain amount of functionality. Most runners have their own queues as subfolders of 'queue'. Each queue consists of message files with times derived from the timestamp of creation and a hash. Each runner processes its queue in order of the timestamps. When its task is complete, it passes the message to the next runner in sequence by moving the file into the appropriate subfolder (with the same filename). By the nature of email, each message is independent of all the others. So we can process them in parallel, as long as there is a way to assign one and only one runner to each message in a queue. The way we do that is to take the top N bits of the hash component of the filename, which we call a slice. Thus each queue has 1 or 2 or 4 or 8, etc slices. To configure multiple slices for the out runner (usually the bottleneck because it's the one that talks almost directly to the network[1]), add [runner.out] instances: 4 to your mailman.cfg and restart. That's what I recommend you do. Footnotes: [1] At least Postfix optimizes relaying by opening a connection to the remote smtpd while still talking to Mailman, and only queues the file on disk if the remote connection fails.

1 month

Re: admin/login/ cannot be accessed

by Abhilash Raj

On Thu, Dec 12, 2019, at 2:37 AM, jean-christophe manciot wrote: > - Django version is 2.2.6-1ubuntu1 > - Disabling HTTP2 in nginx means disabling it for all server blocks listening on the same IP, which would degrade all other servers. > - Doing so leads to another error: > This page isn’t working > <mysite> didn’t send any data. > ERR_EMPTY_RESPONSE > - I cannot run nginx and apache on the same <ip_address>:443 port either. > > I found no error in mailman3 or syslog logs. > In ```/etc/mailman3/mailman.cfg```, I have: > ``` > [logging.debian] > format: %(asctime)s (%(process)d) %(message)s > datefmt: %b %d %H:%M:%S %Y > propagate: no > level: debug > path: mailman.log > ``` > Yet, mailman.log does not seem to show debug level information. mailman.log is only a log file for the Core. Since you are having issues with Django, the requests aren't even touch Mailman Core. Depending on how you are deploying, you should look at mailman-web.log or whatever the name of the log file is. You can find it in the Django settings. > > On Wed, Dec 11, 2019 at 7:04 PM Abhilash Raj <maxking(a)asynchronous.in> wrote: >> >> >> On Wed, Dec 11, 2019, at 9:25 AM, jean-christophe manciot wrote: >> > Ubuntu 20.04 >> > python3-django 2:2.2.6-1ubuntu1 >> > python3-django-hyperkitty 1.3.1 (built from sources) >> > mailman3-full 3.2.2-1 >> >> Which version of Django are you using? >> >> > >> > Nginx server configuration: >> > ``` >> > ... >> > ######## >> > # Static >> > ######## >> > location /favicon.ico >> > { >> > alias <mysite_dir>/static/hyperkitty/img/favicon.ico; >> > } >> > location /static/favicon.ico >> > { >> > alias <mysite_dir>/static/postorius/img/favicon.ico; >> > } >> > location /static/ >> > { >> > alias <mysite_dir>/static/; >> > } >> > >> > ####################### >> > # Upstream uwsgi server >> > ####################### >> > location / >> > { >> > include /etc/nginx/uwsgi_params; >> > uwsgi_pass 127.0.0.1:<uwsgi_server_port>; >> > } >> > ... >> > ``` >> > where: >> > - <mysite_dir> is a symlink to <django_dir>/static >> > - <uwsgi_server_port> matches the one defined in ```/etc/mailman3/uwsgi.ini```: >> > ``` >> > [uwsgi] >> > # Port on which uwsgi will be listening. >> > uwsgi-socket = 127.0.0.1:<uwsgi_server_port> >> > ``` >> > >> >> The config looks good to me in a quick glance. >> >> >> > All 3 systemd services run fine: >> > - mailman3 >> > - mailman3-web >> > - qcluster >> > >> > I'm trying to login to the django administration pages. >> > I get the django administration login page at: >> > https://mysite/admin/login/ >> > Logging in with the admin credentials leads to: >> > ``` >> > This site can’t be reached >> > The webpage at https://mysite/admin/login/ might be temporarily down or >> > it may have moved permanently to a new web address. >> > ERR_HTTP2_PROTOCOL_ERROR >> > ``` >> > This is very strange because it is the URL which I used to get the >> > login page in the first place. >> >> >> Looking at the error, it seems like something somewhere is re-directing to HTTP/2 or the request is based off of HTTP/2 and all the components in the stack don't support HTTP/2, leading to the error message. >> >> I haven't played a lot with HTTP/2 yet so I am not sure which specific component in the stack could be incompatible here. >> >> > >> > If I launch a test web server at another port with: >> > ``` >> > <django_dir># python3 manage.py runserver <mysite_ip_address>:8080 >> > Performing system checks... >> > >> > System check identified no issues (0 silenced). >> > December 11, 2019 - 17:50:48 >> > Django version 2.2.6, using settings 'settings' >> > Starting development server at http://<mysite_ip_address>:8080/ >> > Quit the server with CONTROL-C. >> > ``` >> > and access it at ```http://<mysite_ip_address>:8080/admin/login/``` to >> > login with the same credentials as before, I get through and all the >> > django administration lines appear, although in a degraded layout: >> > ``` >> > Site administration >> > Accounts >> > Email addresses Add Change >> > Authentication and Authorization >> > Groups Add Change >> > Users Add Change >> > Django Mailman 3 >> > Mail domains Add Change >> > Profiles Add Change >> > ... >> > ``` >> > Any idea what could be happening here? >> >> >> Degraded layout is due to missing static files since the development server that you spun off doesn't serve static files. So, that is okay. >> > _______________________________________________ >> > Mailman-users mailing list -- mailman-users(a)mailman3.org >> > To unsubscribe send an email to mailman-users-leave(a)mailman3.org >> > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ >> > >> >> -- >> thanks, >> Abhilash Raj (maxking) > > > -- > Jean-Christophe -- thanks, Abhilash Raj (maxking)

5 years, 2 months

Migrated from MM2 to MM3, all is well except *one* list

by Scott Courtney

Hello, all I have migrated a multi-list server from MM2 to MM3, on Debian 11, and migrated about half a dozen lists with fairly large archives. Absolutely everything has been working for a couple of months, but I am still unable to get one specific list to send mail to its subscribers. Of course, the first place I started was the logs, but that's the problem -- even with logging turned up, it appears to be failing silently as it connects to the MTA. Here's a summary of the environment: OS: Debian Linux 11, 64-bit, on a virtual machine MTA: Exim 4.94.2 List server: Mailman 3.3.3 (installed from Debian repo) Database: MariaDB 10.5.15 All daemons are running on the same host, and communication is either UNIX socket or loopback network interface, for security. Web server: Apache 2.4.53, with proxy to WSGI per the MM3 setup guide Symptom: There are six lists on the server. All six can accept inbound traffic and will post the messages to the (always private, in this environment) archives. The web interface is working perfectly. Five of the lists also can send out messages to subscribers, but the sixth list cannot send traffic to the MTA, or is not attempting to do so. Tests run so far (all resulting in successful posting to archives, but no outbound mail): * Originate mail from inbound SMTP to the list address. * Originate mail from the web UI as a member/moderator. * Originate mail from the web UI as the site admin (also the list owner). * Originate mail from both inbound SMTP and web UI by multiple users. Log data for the inbound traffic (typical test message, some data redacted): 2022-08-27 15:11:57 1oS1Dn-003AWH-Hz <= REDACTED H=REDACTED:25 P=esmtps X=TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128 CV=no K S=2552 DKIM=REDACTED id=REDACTED T="TEST 8.27.001" 2022-08-27 15:11:57 1oS1Dn-003AWH-Hz => MY_LIST(a)lists.MY_DOMAIN R=mailman_main_director T=mailman3_transport H=127.0.0.1 [127.0.0.1] I=[127.0.0.1] C="250 Ok" 2022-08-27 15:11:57 1oS1Dn-003AWH-Hz Completed Log data for MM3 smtp.log (again, partially redacted): Aug 27 15:11:57 2022 (690182) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Aug 27 15:11:57 2022 (690182) Peer: ('127.0.0.1', 48170) Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) handling connection Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'LHLO REDACTED' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'MAIL FROM:<REDACTED>' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) sender: REDACTED Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'RCPT TO:<erevnite(a)lists.4th.com>' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) recip: MY_LIST(a)lists.MY_DOMAIN Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'DATA' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'QUIT' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) connection lost Aug 27 15:11:57 2022 (690182) Connection lost during _handle_client() Originally I thought the "connection lost" indicated a failure, but that is happening for all the lists, and also on successful traffic. Since it occurs after the QUIT command in the SMTP session, and based on some online searching, my current understanding is that this is harmless and simply indicates the MTA closed the socket after the QUIT. (Please correct me if that is wrong.) Log data for mailman.log, with partial redaction: Aug 27 15:11:57 2022 (690181) ACCEPT: <REDACTED> Aug 27 15:11:59 2022 (690178) HyperKitty archived message <REDACTED> to https://lists.MY_DOMAIN/hyperkitty/list/MY_LIST@lists.MY_DOMAIN/message/G3D… 19:11:59 [Q] INFO Enqueued 518 19:11:59 [Q] INFO Enqueued 519 19:11:59 [Q] INFO Enqueued 520 19:11:59 [Q] INFO Enqueued 521 19:11:59 [Q] INFO Enqueued 522 19:11:59 [Q] INFO Enqueued 523 19:11:59 [Q] INFO Enqueued 524 INFO 2022-08-27 19:11:59,648 690164 hyperkitty.views.mailman Archived message <REDACTED> to https://lists.MY_DOMAIN/hyperkitty/list/MY_LIST@lists.MY_DOMAIN/message/G3D… [pid: 690164|app: 0|req: 4688541/4688541] 2600:3c04::f03c:91ff:fe02:9cc1 () {64 vars in 1255 bytes} [Sat Aug 27 19:11:59 2022] POST /hyperkitty/api/mailman/archive?key=xombsjgLDqe6d1oxvBaQWH4ZT81uCEjH => generated 113 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 160 bytes (1 switches on core 0) 19:12:00 [Q] INFO Process-1:4 processing [update_from_mailman] 19:12:00 [Q] INFO Process-1:3 processing [sender_mailman_id] 19:12:00 [Q] INFO Process-1:2 processing [rebuild_thread_cache_new_email] 19:12:00 [Q] INFO Process-1:1 processing [compute_thread_positions] 19:12:00 [Q] INFO Processed [rebuild_thread_cache_new_email] 19:12:00 [Q] INFO Process-1:2 processing [rebuild_mailinglist_cache_recent] 19:12:00 [Q] INFO Processed [sender_mailman_id] 19:12:00 [Q] INFO Process-1:3 processing [rebuild_mailinglist_cache_for_month] 19:12:00 [Q] INFO Process-1:1 processing [check_orphans] 19:12:00 [Q] INFO Processed [compute_thread_positions] 19:12:00 [Q] INFO Processed [rebuild_mailinglist_cache_for_month] 19:12:00 [Q] INFO Processed [check_orphans] 19:12:00 [Q] INFO Processed [rebuild_mailinglist_cache_recent] 19:12:00 [Q] INFO Processed [update_from_mailman] [pid: 690164|app: 0|req: 4693691/4693691] 184.94.240.88 () {60 vars in 929 bytes} [Sat Aug 27 19:21:50 2022] HEAD / => generated 0 bytes in 2 msecs (HTTP/1.1 301) 6 headers in 204 bytes (1 switches on core 0) [pid: 690164|app: 0|req: 4693694/4693694] 184.94.240.88 () {60 vars in 1009 bytes} [Sat Aug 27 19:21:50 2022] HEAD /postorius/lists/ => generated 7136 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 163 bytes (1 switches on core 1) There is nothing at all in the Exim log indicating an outbound transaction was attempted by Mailman. I know Exim and Mailman are communicating because all the lists use the same SMTP configuration for Exim and Mailman (i.e., the non-working list is not in any way exceptional). I've done a lot of searching online to find a solution, and I did see that a couple of others had asked about this type of problem, but the only one I found that got a response had an issue that didn't apply in my small environment. Absent any other ideas, I'm going to try inserting some trace logging into the Python code, but I can't figure out what's wrong here. I've even gone to the point of looking at the SQL rows to verify this list is exactly parallel to another working list, except its name and integer ID numbers for various unique auto-increment keys. Any suggestions are appreciated. Thanks!

2 years, 6 months

Re: Time Stand Still

by Fabian A. Santiago

On November 23, 2017 11:38:00 AM EST, Barry Warsaw <barry(a)list.org> wrote: >Somewhen in the dark recesses of intarweb history, I found myself as >the project leader for both Jython (née JPython) and GNU Mailman. I'd >been involved with Jython since it was invented by Jim Hugunin around >the time he came to work with us at Pythonlabs. I'd been contributing >to Mailman since we inherited John Viega's Python-based Dave Matthews >Band list server, and put it to use replacing python.org's Majordomo >installation. > >I'd enjoyed both projects, but knew I could not lead both, so I had to >make a choice. I chose to turn over Jython to a team that's done a >much better job over the years than I ever could have. Something about >email, and especially the communication and collaboration patterns that >it facilitates, really fascinated me. I know, I know, but we all have >our lapses of sanity. Mine has lasted almost 20 years, a bit more than >"momentary" perhaps. > >I've rarely gotten paid to work on Mailman, but it did provide me some >great opportunities. Most notably it led to my 10 year stint at >Canonical. I was originally hired on there to integrate mailing lists >with Launchpad, and Mailman was the obvious choice. I learned a ton >doing that project, and working within the constraints of integrating >the two Python-based systems, especially since Launchpad was originally >not free software and Mailman was GPL'd. Later, the Zope-based >Launchpad source code was released under the AGPL, making much of the >monkeypatching unnecessary, but by then the system was solid and >reliable, and you don't fix what's not broken. > >Except, I guess I did. I took a lot of the lessons from that work, >along with a good hard look at all the problems with Mailman 2, and >began to break another cardinal rule of software development: second >system syndrome. The result is Mailman 3. It took forever, and we're >still not at complete feature parity with Mailman 2, but at least it's >Real Enough to be used at many Real Sites, including python.org and >lists.fedoraprojects.org. > >It would be ridiculous for me to take significant credit for this. I >have to acknowledge the amazing user community -- you! -- for all the >support, patches, suggestions, feedback, patience, criticism, >donations, and contributions that you've given to the project, and to >me personally over the years. And my deepest gratitude goes to all the >core developers that have stayed or come and gone, but most especially >the current Cabal: Abhilash Raj, Aurelien Bompard, Florian Fuchs, Mark >Sapiro, Stephen J. Turnbull, Terri Oda. You should know that each and >every one of them is truly awesome, both in what they contribute >technically, and in their amazing friendships. Mailman is infinitely >better because of their involvement, and I've loved spending time with >them over the years at the Pycon sprints, making releases and sharing >teas and meals. > >My blog is called We Fear Change, and that's humorously taken from a >90's bit in Mike Myer's excellent Wayne's World movie (a phrase >actually uttered by the brilliant Dana Carvey as Garth). The irony of >course is that while we all may fear change, it's the one constant >thing we can count on. And in fact, we *require* change to thrive, >because if you aren't changing, you aren't alive. Time, and being >engaged with life's vagaries, means there's no alternative to change; >it must be embraced. > >And so, with a vague reference to the many (good!) changes in my >personal and professional life, I'm announcing that I'm stepping down >from the project leadership role of GNU Mailman, effective... nowish! >And it's with unanimous agreement among the GNU Mailman Steering >Committee (a.k.a. the Mailman Cabal), that we are announcing Abhilash >Raj as the new project leader. > >If you don't recognize Abhilash's name, you probably aren't paying >attention, at least to Mailman 3. Abhilash came to us in 2013 as a >Google Summer of Code student, and he's become one of the project's >most valuable contributors. His list of accomplishments is long, and >it includes everything from redesigning the website, to integrating CI >with our GitLab build system, porting our code to the SQLAlchemy ORM, >adding MySQL support, revving up adoption through his Docker images, >along with his great coding work on Core, Postorius, HyperKitty, and >mailmanclient. > >This transition is good for the project too. Email, its defining >protocols and standards, and its role in our daily lives, has changed >profoundly since the early days of Mailman. A fresh perspective and >enthusiasm will help keep Mailman relevant to the changing ways we -- >especially the FLOSS and tech communities -- communicate. > >Please join me in supporting Abhilash in every way possible as he takes >over in this new role as project leader. I'll be here when and if >needed, even as I create space in my "spare" time for... Something >Else. I look forward to the vision that Abhilash will bring to the >project, and I know that he will do a great job. To me, Mailman has >always been about collaboration, and the best >way for it to succeed is for you to continue to contribute your >insights, experiences, opinions, and skills with positive intention. > >-Barry > >https://www.wefearchange.org/ Congrats Abhilash! Love the docker images! Thank you Barry! Mailman rocks! -- Thanks, Fabian S. OpenPGP: 3C3FA072ACCB7AC5DB0F723455502B0EEB9070FC

7 years, 3 months

Re: How To Install Mailman 3 on Debian 10 (Complete Guide)

by Brian Carpenter

On 2/25/21 10:35 PM, Abhilash Raj wrote: > I think it is okay to be pick up a specific configuration and provide a > guide to installation keeping that as the target technology. As it happens > to be, the official doc uses the same set, Nginx, Postfix and PostgSQL. > > https://docs.mailman3.org/en/latest/install/virtualenv.html Thank you for pointing the above out. I will keep track of the URLs you provided in this thread and I am willing to also assist to try to make sure such links are propagated via search engine results. > I don't see anything very specific to Debian in the doc except the package > names and I think they are similar enough in Debian derivative distros that > we can write up that assumption up front in the top. > > Given that most of the installation is based off on pip, I don't think there is > going to be a *lot* of deviation for even RHEL/Fedora based distros when it > comes to Mailman itself. Some minute deviations could occur with the > behavior of certain commands, like `nano` doesn't exist, but I still expect > 95% of the doc to be the same, except obviously the package names. I agree. One of my goals was to make it easy for Mailman 3 adopters to use a different installation method other than a repo since distro packages tend to install older versions of Mailman 3 core, Postorius, and Hyperkitty. At this stage of Mailman 3 development, these new versions bring a LOT of improvements to the Mailman 3 user. I am also willing to start testing out my installation method on other distros and document the differences. If you are willing to help me, I have no problem posting installation instructions and tips to Mailman 3's official documentation. I am also more than willing to keep such documentation up to date. I am always being called upon to install Mailman 3 for someone, so I seem to be on the front-line when it comes to "surprises" such as the issue with the cryptography module. That puts me in a good position to update documentation simply because I need to know such things for the next install. > I am not that well versed in SEO, but the official website https://list.org > does point to it. Maybe not prominently enough for new users to discover? > I'll see if we can make the discovery of docs.mailman3.org better. I think the issue I found was running into outdated documentation for older versions of Mailman 3. Even doing a search for an up to date changelog or release notes are problematic. Again, I am also willing to help out with trying to get such vital pieces of information showing up prominently in search results. > I guess it is fine to be opinionated about choosing the technologies but > yet provide pointers to help people using other technologies. > > Right now, I do that using "Hints" and "Notes" and "See also" banners > through out the documentation to point to different pages where they > can lookup setup for other technologies. For example, how to setup > MySQL if official doc recommends Postgres. Are there still any outstanding issues with using MySQL? I know I ran into some when I tried to use MySQL and chose to go with PostgreSQL (which I am so glad I did). I still recall even seeing some recent issues brought to this list for someone using MySQL. I know MySQL is very popular for websites so I do believe the momentum for someone to set up a Mailman 3 server is to go with MySQL. Perhaps just putting up some heads up warnings (here there be dragons!) IF there are still issues out in the wild for those wanting to use MySQL with Mailman 3. Personally I will always steer clients to PostgreSQL when I get a chance. My own documentation reflects that bias. > Gitlab has a fairly good experience editing REsT files and you don't > necessarily > have to do much outside of a Web Browser actually. It lets you preview before > saving like wiki.list.org and you can submit it for review and merging into > main repo. > > Would you be more willing to contribute to official documentation if we > documented the Gitlab contribution process better? Absolutely. My current installation documentation was the result of a certain beautiful Mailman developer who I won't name (just protecting your privacy Mark!) providing me assistance when I decided to go with the certain installation direction. So if you scratch my back (i.e. teach me Master for I am your student!) then I will scratch yours. > Because we haven't yet gotten to updating those in the official docs, > unfortunately. And there are more things missing like Fulltext search (Xapian > recommended) and more of the Email setup like SPF and DKIM. Very much > aligned with your plans, which is why I suggested contributing updates > to docs.mailman3.org instead :) Well show me how to update the documentation and I will certainly take care of those things. I am also now comfortable in getting OpenDKIM working with Mailman 3 so I can also provide documentation there as well. I have done Xapian installations for clients but unfortunately did not document my methods so I will need to work on that. How do you feel about documentation for the use of Elasticsearch and Solr? Remember, I was a Mailman 2 host for years using cPanel. cPanel took the thinking of out a lot of things when it came to Mailman 2 hosting and I don't have that crutch anymore with Mailman 3. -- Brian Carpenter Harmonylists.com Emwd.com

4 years

Re: Basic settings for a discussion list

by Abhilash Raj

On Sat, Aug 31, 2019, at 9:25 PM, paul(a)arenson.org wrote: > Over the last 20 hours I have been trying to figure out setup for list > members to sign up. One person has kindly helped out, though I am > still unsure of the difference between the several signups. That > thread is here and I welcome more advice. > > https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/… > > But it occurs to me that my problem is that I have not actually > configured my list. So this thread is for help on that. I have found a > guide for general users here: > http://docs.mailman3.org/en/latest/userguide.html > > But I have only found a highly technical guide for administrators here: > https://docs.mailman3.org/projects/mailmanclient/en/latest/src/mailmanclien… > > It is written for coders and I am looking for a guide similar to the > one for users. It seems not to exist yet. > > WHAT I WANT > Discussion only list. Ideally anyone who signs up via the various ways > should be able to get the archives and post via email and the interface > if they want. The list should be closed to others. > > In the absence of any guide, all I can do is cross check with you guys > and hope I am on the right track. I am non technical, although I > managed a Mailman 2 list without difficulty. > > > 1) FIRST ANOMALY > Under https://list.tokyoprogressive.org/postorius/accounts/subscriptions/ > I see myself twice: > > List Name Subscription Address Delivery Mode Role > discuss.list.tokyoprogressive.org paul(a)tokyoprogressive.org None member > discuss.list.tokyoprogressive.org paul(a)tokyoprogressive.org regular owner > > Is this correct? I added My name in one field somewhere and this is the > result. I even got a welcome message. > I think it is because I subscribed to my own list.If this was not > necessary, what should I do? No, this isn't anomaly. Check out the "Role" column. You are an "owner" of the list and are thus allowed to change the list settings. You are also a "member", which means you will get emails from this mailing list. > > Global based preferences are listed. Address and email based ones are > not showing either enabled or disabled. They are unset, which means they are set to whatever the default value in the backend is. I know this is terrible in many regards, but we have an issue open for this. Hopefully, this will improve in future releases. > > 2) MANAGE LISTS > > For myself I have done the following in my MAILING LIST SETTINGS > > Subscribe to this list > My email and name > (Maybe that is why i am listed twice) That's right. > > Now I see > > 3) Users > For now only me- I am listed as member and owner. > There will be no other moderators or owners > I have no idea what a non member is (if they register aren't they a member?) I explained non-member in other email you sent. > > 4) Templates > This is showing nothing at this point. Before it was. I am not sure > what i did to stop it from showing. Perhaps because I made it a private > list? If this is a new list, it will probably not have any templates. > > 5) Settings > a) List identity > I have it set for NOT SHOWING on the Index page because I will link to > it from my news magazine website as a way to comment on articles. Is > this ok? Other data will be filled in later. I think I am confusing > this with private list. I also will have no other list. > > b) AUTO RESPOND LIST OWNER-- I added a short note thanking people. > Respond and Continue Processing > > c) AUTO RESPOND POSTER--Your message has been been received. If you > are a member, it should be posted soon. Respond and Continue Processing > > d) AUTO RESPOND REQUESTS--No auto response > OTHER THINGS DEFAULT > > e) URL WELCOME MESSAGE--NOT SURE. I want people to subscribe > themselves but I also have 500 people from another list from DADA I > will import. I want them automatically be able to post. Is it > possible? If so, I hope to add them manually. DO I compose a text on > my website and welcome them by adding the URL, or enter it in my list? This option is deprecated, and actually now removed. > > f) NOTIFY OF MEMBERSHIP CHANGES--Yes > > g) ALTER MESSAGES ---I have left most things as they are. However, as > this is a discussion list, I want replies ONLY to go to the list. I > chose that. > > h) DMARC --I have left as is > > i) DIGEST--I have left as is > > > 6) MASS OPERATIONS--I have left as is > > 7) BAN--left as is > > 8) MESSAGE ACCEPTANCE--I have left as is > > > 9) HEADER FILTERS--left as is > > 10) ARCHIVES--Private Archives (Maybe this is why templates do not show???) > > 11) SUBSCRIPTION POLICY--MODERATE-- I intend to send them an email to > confirm their reasons for subscribing to avoid spammers. > > LAST THING. My Gravity does not work. I edited it online as per > instructions but it does not appear. > > That is all I can think of. > > Thanks in advance. > > paul > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- thanks, Abhilash Raj (maxking)

5 years, 6 months

Re: Hyperkitty CPU usage

by Alain Kohli

I have not seen any improvement over time, so I switched the backend to Xapian. That works like a charm now, I guess our lists were indeed just too busy for whoosh. On 5/7/19 5:15 PM, Alain Kohli wrote: > Well, I understand that it might take a while, but the process is > running for over 200h now, I doubt it can really take _that_ long if > everything is working properly. Especially because the rebuild_index > command is only taking a few hours at most, doesn't that essentially > do the same? > > We have ~700k messages in ~300 mailing lists in case that helps you > estimate how long it could take. I can take a look at alternative > backends, though. > > > On 5/7/19 4:32 PM, Abhilash Raj wrote: >> On Sat, Apr 27, 2019, at 6:22 PM, Alain Kohli wrote: >>> I'm running a custom image which is based on an older version of the >>> one >>> here: https://github.com/maxking/docker-mailman. I attached it below. >>> But I separated postorius and hyperkitty, so hyperkitty is running in >>> its own container. I'm deploying the image with a plain 'docker run' >>> behind nginx. I made fulltext_index persistent now, but it didn't get >>> populated with anything yet. I don't really have an error traceback >>> because there is never an error thrown. The only thing with some >>> content >>> is uwsgi-error.log, which you can find below. I'm also still getting >>> the >>> "A string literal cannot contain NUL (0x00) characters." messages. I >>> also noticed that it takes incredibly long for the webinterface to load >>> (several minutes) even though there doesn't seem to be any process >>> consuming notable resources apart from the minutely job. >>> >>> Funnily enough, I have the exact same image deployed on a second server >>> as well for testing. On that one everything works fine. The only >>> difference is that on the problematic one I have a lot more mailing >>> lists/archives and that I imported them from mailman2. Could something >>> have gone wrong during the import? I used the regular hyperkitty_import >>> command. >> Yes, this is because `whoosh`, the library set by default to run >> fulltext >> indexing is a pure python implementation and quite slow in busy lists. >> >> We do support more backends though, see [1] for a list of all the >> supported >> search backends. Something like Xapian(C++) or Elasticsearch/Solr(Java) >> should be much better in terms of performance. >> >> [1]: >> https://django-haystack.readthedocs.io/en/master/backend_support.html >> >>> uwsgi-error.log: >>> >>> *** Starting uWSGI 2.0.18 (64bit) on [Sat Apr 27 22:50:17 2019] >>> *** >>> compiled with version: 6.4.0 on 27 April 2019 22:48:42 >>> os: Linux-4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) >>> nodename: hyperkitty.docker >>> machine: x86_64 >>> clock source: unix >>> detected number of CPU cores: 4 >>> current working directory: /home/hyperkitty >>> detected binary path: /usr/local/bin/uwsgi >>> !!! no internal routing support, rebuild with pcre support !!! >>> setgid() to 82 >>> setuid() to 82 >>> chdir() to /home/hyperkitty >>> your memory page size is 4096 bytes >>> detected max file descriptor number: 1048576 >>> lock engine: pthread robust mutexes >>> thunder lock: disabled (you can enable it with --thunder-lock) >>> uwsgi socket 0 bound to TCP address 0.0.0.0:8081 fd 8 >>> uwsgi socket 1 bound to TCP address 0.0.0.0:8080 fd 9 >>> Python version: 3.6.8 (default, Jan 30 2019, 23:54:38) [GCC 6.4.0] >>> Python main interpreter initialized at 0x55dfaa41c980 >>> python threads support enabled >>> your server socket listen backlog is limited to 100 connections >>> your mercy for graceful operations on workers is 60 seconds >>> [uwsgi-cron] command "./manage.py runjobs minutely" registered as >>> cron task >>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" >>> registered >>> as cron task >>> [uwsgi-cron] command "./manage.py runjobs hourly" registered as >>> cron >>> task >>> [uwsgi-cron] command "./manage.py runjobs daily" registered as >>> cron task >>> [uwsgi-cron] command "./manage.py runjobs monthly" registered as >>> cron task >>> [uwsgi-cron] command "./manage.py runjobs weekly" registered as >>> cron >>> task >>> [uwsgi-cron] command "./manage.py runjobs yearly" registered as >>> cron >>> task >>> mapped 208576 bytes (203 KB) for 4 cores >>> *** Operational MODE: threaded *** >>> WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter >>> 0x55dfaa41c980 pid: 1 (default app) >>> *** uWSGI is running in multiple interpreter mode *** >>> spawned uWSGI master process (pid: 1) >>> spawned uWSGI worker 1 (pid: 40, cores: 4) >>> Sat Apr 27 22:50:18 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> minutely" (pid 45) >>> [uwsgi-daemons] spawning "./manage.py qcluster" (uid: 82 gid: 82) >>> 22:50:21 [Q] INFO Q Cluster-47 starting. >>> 22:50:21 [Q] INFO Process-1:1 ready for work at 59 >>> 22:50:21 [Q] INFO Process-1:2 ready for work at 60 >>> 22:50:21 [Q] INFO Process-1:3 ready for work at 61 >>> 22:50:21 [Q] INFO Process-1:4 ready for work at 62 >>> 22:50:21 [Q] INFO Process-1:5 monitoring at 63 >>> 22:50:21 [Q] INFO Process-1 guarding cluster at 58 >>> 22:50:21 [Q] INFO Process-1:6 pushing tasks at 64 >>> 22:50:21 [Q] INFO Q Cluster-47 running. >>> 22:59:31 [Q] INFO Enqueued 3403 >>> 22:59:31 [Q] INFO Process-1:1 processing [update_from_mailman] >>> 22:59:33 [Q] INFO Processed [update_from_mailman] >>> Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> quarter_hourly" (pid 73) >>> Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> hourly" (pid 74) >>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running >>> with pid 73 exited after 64 second(s) >>> 23:01:28 [Q] INFO Enqueued 3404 >>> 23:01:29 [Q] INFO Process-1:2 processing >>> [rebuild_mailinglist_cache_recent] >>> [uwsgi-cron] command "./manage.py runjobs hourly" running with pid >>> 74 exited after 91 second(s) >>> Sat Apr 27 23:01:36 2019 - uwsgi_response_write_body_do(): Broken >>> pipe [core/writer.c line 341] during GET / (212.203.58.154) >>> OSError: write error >>> 23:01:36 [Q] INFO Processed [rebuild_mailinglist_cache_recent] >>> Sat Apr 27 23:15:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> quarter_hourly" (pid 88) >>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running >>> with pid 88 exited after 4 second(s) >>> 23:28:24 [Q] INFO Enqueued 3405 >>> 23:28:24 [Q] INFO Process-1:3 processing [update_from_mailman] >>> 23:28:25 [Q] INFO Processed [update_from_mailman] >>> Sat Apr 27 23:30:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> quarter_hourly" (pid 96) >>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running >>> with pid 96 exited after 4 second(s) >>> 23:44:40 [Q] INFO Enqueued 3406 >>> 23:44:40 [Q] INFO Process-1:4 processing [update_from_mailman] >>> 23:44:41 [Q] INFO Processed [update_from_mailman] >>> Sat Apr 27 23:45:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> quarter_hourly" (pid 104) >>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running >>> with pid 104 exited after 4 second(s) >>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> quarter_hourly" (pid 113) >>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> hourly" (pid 114) >>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> daily" (pid 115) >>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py >>> runjobs >>> weekly" (pid 116) >>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running >>> with pid 113 exited after 55 second(s) >>> [uwsgi-cron] command "./manage.py runjobs weekly" running with pid >>> 116 exited after 55 second(s) >>> 00:01:36 [Q] INFO Enqueued 3407 >>> 00:01:36 [Q] INFO Process-1:1 processing >>> [rebuild_mailinglist_cache_recent] >>> [uwsgi-cron] command "./manage.py runjobs hourly" running with pid >>> 114 exited after 99 second(s) >>> 00:01:50 [Q] INFO Processed [rebuild_mailinglist_cache_recent] >>> 00:04:52 [Q] INFO Enqueued 3408 >>> 00:04:52 [Q] INFO Process-1:2 processing [update_from_mailman] >>> 00:04:54 [Q] INFO Processed [update_from_mailman] >>> >>> Dockerfile: >>> >>> FROM python:3.6-alpine3.7 # Add startup script to container COPY >>> assets/docker-entrypoint.sh /usr/local/bin/ # Install packages and >>> dependencies for hyperkitty and add user for executing apps. # >>> It's >>> important that the user has the UID/GID 82 so nginx can access the >>> files. RUN set -ex \&& apk add --no-cache --virtual .build-deps >>> gcc >>> libc-dev linux-headers git \postgresql-dev \&& apk add --no-cache >>> --virtual .mailman-rundeps bash sassc mailcap \postgresql-client >>> curl \&& pip install -U django==2.2 \&& pip install >>> git+https://gitlab.com/eestec/mailmanclient >>> \git+https://gitlab.com/mailman/hyperkitty@c9fa4d4bfc295438d3e01cd93090064d004cf44d >>> \git+https://gitlab.com/eestec/django-mailman3 \whoosh \uwsgi >>> \psycopg2 \dj-database-url \typing \&& apk del .build-deps \&& >>> addgroup -S -g 82 hyperkitty \&& adduser -S -u 82 -G hyperkitty >>> hyperkitty \&& chmod u+x /usr/local/bin/docker-entrypoint.sh# Add >>> needed files for uwsgi server + settings for django COPY >>> assets/__init__.py /home/hyperkittyCOPY assets/manage.py >>> /home/hyperkittyCOPY assets/urls.py /home/hyperkittyCOPY >>> assets/wsgi.py /home/hyperkittyCOPY assets/uwsgi.ini >>> /home/hyperkittyCOPY assets/settings.py /home/hyperkitty# Change >>> ownership for uwsgi+django files and set execution rights for >>> management script RUN chown -R hyperkitty /home/hyperkitty && >>> chmod >>> u+x /home/hyperkitty/manage.py# Make sure we are in the correct >>> working dir WORKDIR /home/hyperkittyEXPOSE 8080 8081# Use stop >>> signal for uwsgi server STOPSIGNAL SIGINTENTRYPOINT >>> ["docker-entrypoint.sh"]CMD ["uwsgi", "--ini", >>> "/home/hyperkitty/uwsgi.ini"] >>> >>> On 4/27/19 7:58 PM, Abhilash Raj wrote: >>>> On Sat, Apr 27, 2019, at 9:40 AM, Alain Kohli wrote: >>>>> I have run "python manage.py rebuild_index" before, doesn't that do >>>>> clear_index as well? Apart from that, I run hyperkitty in a docker >>>>> container and didn't know fulltext_index should be persistent, so >>>>> that >>>>> got deleted after every version update for sure. >>>> Which images are you using and how are you deploying them? >>>> >>>> You should persist fulltext_index, yes, and possibly logs if you need >>>> them for debugging later. >>>> >>>> Can you paste the entire error traceback? >>>> >>>>> On 4/26/19 10:18 PM, Mark Sapiro wrote: >>>>>> On 4/26/19 11:14 AM, Alain Kohli wrote: >>>>>>> I see loads of "A string literal cannot contain NUL (0x00) >>>>>>> characters." >>>>>>> messages, but I haven't found missing messages in the archives >>>>>>> yet. Not >>>>>>> sure how that could be related, though. Apart from that I don't see >>>>>>> anything unusual. The other jobs (quarter_hourly, hourly, etc.) >>>>>>> seem to >>>>>>> run and finish normally. >>>>>> Did you upgrade from a Python 2.7 version of HyperKitty to a >>>>>> Python 3 >>>>>> version? The Haystack/Whoosh search engine databases are not >>>>>> compatible >>>>>> between the two and "A string literal cannot contain NUL (0x00) >>>>>> characters." is the symptom. >>>>>> >>>>>> You need to run 'python manage.py clear_index' or just remove all >>>>>> the >>>>>> files from the directory defined as 'PATH' under >>>>>> HAYSTACK_CONNECTIONS in >>>>>> your settings file (normally 'fulltext_index' in the same >>>>>> directory that >>>>>> contains your settings.py. >>>>>> >>>>> _______________________________________________ >>>>> Mailman-users mailing list -- mailman-users(a)mailman3.org >>>>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org >>>>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ >>>>> >>> _______________________________________________ >>> Mailman-users mailing list -- mailman-users(a)mailman3.org >>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org >>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ >>> >

5 years, 9 months

Re: config incoming email in my domain cpanel

by Rafael Mora

El jue., 3 ago. 2017 a las 16:07, tlhackque via Mailman-users (< mailman-users(a)mailman3.org>) escribió: > You've said you're working with labbrands.com. > > labbrands.com has address 192.185.51.89 > mail.labbrands.com has address 192.185.51.89 > www.labbrands.com is an alias for labbrands.com. > > telnet proves that Exim is running on that IP address. Either you're > running it, or hostgator is redirecting the SMTP ports. > > You added: > lists.labbrands.com has address 190.145.27.66 > > Presumably that's a different machine. > > So that now needs an MX record, and a firewall routing. I added the MX record on our hostgator hosting like this: [image: image.png] Is it correct??? > And the SPF > records ... and spam and virus filters, and all the other stuff you need > to setup an independent, public-facing smtp server. > > But at least you won't have to relay. > > Good luck. > > On 03-Aug-17 16:55, Rafael Mora wrote: > > El jue., 3 ago. 2017 a las 15:49, tlhackque via Mailman-users (< > > mailman-users(a)mailman3.org>) escribió: > > > >> I think you're confused. > >> > >> You already have mail.labbrands.com set up as the MX record for > >> labbrands.com. And it has an A record with the same address as your > >> webserver. > >> > > I'm working with the hostgator mailserver, we are not running a local > > mailserver. > > > > As suggested I added an A record like this: > > [image: image.png] > > > > Is it correct? is it redirecting to my Ip so I can redirect it to my > > postfix/mm3 server? > > > > > >> So if you're getting e-mail on that domain, there's another e-mail > >> server running on that IP address. You can't have 2 servers on port > >> 25. In that case, as has been noted before, you'll need to setup a > >> relay in that server, not a firewall redirect. Depending on your MTA, > >> you would need to relay to your internal server. And make sure your > >> firewall setup allows your MTA to do this. > >> > >> We can see it's EXIM: > >> > >> telnet mail.labbrands.com 25 > >> Trying 192.185.51.89... > >> Connected to mail.labbrands.com (192.185.51.89). > >> Escape character is '^]'. > >> help > >> 220-gator4137.hostgator.com ESMTP Exim 4.87 #1 Thu, 03 Aug 2017 > 15:47:48 > >> -0500 > >> 220-We do not authorize the use of this system to transport unsolicited, > >> 220 and/or bulk e-mail. > >> 214-Commands supported: > >> 214 AUTH STARTTLS HELO EHLO MAIL RCPT DATA NOOP QUIT RSET HELP > >> quit > >> 221 gator4137.hostgator.com closing connection > >> Connection closed by foreign host. > >> > >> Or, consolidate all your e-mail to one server - which is a lot easier to > >> manage unless you have a really big operation. Postfix is probably the > >> right choice, but requires more setup. > >> > >> Although Mailman3 configuration is not well documented (as you've > >> discovered), you may want to get help from someone with more general > >> network and mail experience. You're now into territory that is, as > >> Simon indicated, not Mailman-specific. > >> > >> On 03-Aug-17 16:28, Rafael Mora wrote: > >>> El jue., 3 ago. 2017 a las 15:26, Mark Sapiro (<mark(a)msapiro.net>) > >> escribió: > >>>> On 08/03/2017 01:22 PM, Rafael Mora wrote: > >>>>> El jue., 3 ago. 2017 a las 15:18, Mark Sapiro (<mark(a)msapiro.net>) > >>>> escribió: > >>>>>> You need to forward port 25 for SMTP mail delivery and if you want > the > >>>>>> web UI (Postorius and HyperKitty) accessible from the outside, port > 80 > >>>>>> for http and/or port 443 for https > >>>>>> < > >>>>>> > >> > https://www.iana.org/assignments/service-names-port-numbers/service-names-p… > >>>>>>> . > >>>>> I mean for incoming mail redirected from my hostgator hosting, > because > >>>> when > >>>>> I suscribe an email address MM3 sends a confirmation email, so I have > >> to > >>>>> reply to be suscribed to the list. > >>>> As I said, for mail delivery you need to forward port 25 to the > Mailman > >>>> server. > >>>> > >>> Ok so I'll redirect in my zentyal firewall the port 25 to my local > >> centos7 > >>> with postfix and MM3 with IP 192.168.1.42. Thanks Mark. > >>> > >>> > >>>> -- > >>>> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > >>>> San Francisco Bay Area, California better use your sense - B. Dylan > >>>> _______________________________________________ > >>>> Mailman-users mailing list > >>>> mailman-users(a)mailman3.org > >>>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > >>>> > >> -- > >> This communication may not represent my employer's views, > >> if any, on the matters discussed. > >> > >> _______________________________________________ > >> Mailman-users mailing list > >> mailman-users(a)mailman3.org > >> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > >> > > -- > This communication may not represent my employer's views, > if any, on the matters discussed. > > _______________________________________________ > Mailman-users mailing list > mailman-users(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- Atentamente / Best Regards Ing. Rafael Mora

7 years, 7 months

Re: new list not working

by Christian

I am only having issues with the new list. Below are logfile entries Smtp.log ++++++ Mar 09 20:57:09 2023 (9401) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Mar 09 20:57:09 2023 (9401) Peer: ('127.0.0.1', 35982) Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) handling connection Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'LHLO zarathustra.ccalternatives.org' Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'MAIL FROM:<christian(a)naturalintelligence.us> SIZE=3653' Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) sender: christian(a)naturalintelligence.us Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'RCPT TO:<redlatinleg(a)lists.ccalternatives.org>' Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) recip: redlatinleg(a)lists.ccalternatives.org Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'DATA' Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'QUIT' Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) connection lost Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) Connection lost during _handle_client() Access.log ++++++++ 127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/recent-threads HTTP/1.0" 200 4676 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" 127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/favorites HTTP/1.0" 200 58 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" 127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/top-threads HTTP/1.0" 200 4676 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" 127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/posted-to HTTP/1.0" 200 54 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" Mailman.log ++++++++ Mar 09 20:57:09 2023 (9400) ACCEPT: <032101d952c9$caf09990$60d1ccb0$(a)naturalintelligence.us> Mar 09 13:03:42 2023 (9397) HyperKitty archived message <032101d952c9$caf09990$60d1ccb0$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat… [09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg(a)lists.ccalternatives.org HTTP/1.1" 200 497 "-" "GNU Mailman REST client v3.3.3" [09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg(a)lists.ccalternatives.org/config HTTP/1.1" 200 3805 "-" "GNU Mailman REST client v3.3.3" [09/Mar/2023:21:03:42 +0000] "GET /3.1/addresses/cs(a)safecomputing.org HTTP/1.1" 200 402 "-" "GNU Mailman REST client v3.3.3" [09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg.lists.ccalternatives.org/roster/owner HTTP/1.1" 200 665 "-" "GNU Mailman REST client v3.3.3" [09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg.lists.ccalternatives.org/roster/moderator HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.3" [09/Mar/2023:21:03:42 +0000] "GET /3.1/users/christian(a)naturalintelligence.us HTTP/1.1" 200 297 "-" "GNU Mailman REST client v3.3.3" [09/Mar/2023:21:08:22 +0000] "GET /3.1/users/58f1e71ff425473d935fda2a7f9c2171 HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.3" mailmansuite.log ++++++++++++ File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django_mailman3/lib/paginator.py", line 56, in count return self.function(count=0, page=1).total_size File "/opt/mailman/mm/venv/lib/python3.7/site-packages/postorius/views/list.py", line 962, in _get_list_page advertised=advertised, mail_host=mail_host, count=count, page=page) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/client.py", line 210, in get_list_page return Page(self._connection, url, MailingList, count, page) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/page.py", line 37, in __init__ self._create_page() File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/page.py", line 62, in _create_page response, content = self._connection.call(self._build_url()) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/connection.py", line 169, in call 'Could not connect to Mailman API: ', repr(e)) mailmanclient.restbase.connection.MailmanConnectionError: ('Could not connect to Mailman API: ', 'ConnectionError(MaxRetryError("HTTPConnectionPool(host=\'localhost\', port=8001): Max retries exceeded with url: /3.1/lists?advertised=true&count=0&page=1 (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f910b1d7da0>: Failed to establish a new connection: [Errno 111] Connection refused\'))"))') ERROR 2023-03-08 13:52:33,078 623 django.request Service Unavailable: /mailman3/lists/ ERROR 2023-03-08 13:52:33,078 623 django.request Service Unavailable: /mailman3/lists/ INFO 2023-03-08 14:12:32,804 5421 hyperkitty.views.mailman Archived message <01e001d9520a$3ca64320$b5f2c960$(a)sewagesludgeactionnetwork.com> to https://lists.ccalternatives.org/archives/list/ssan@lists.ccalternatives.or… INFO 2023-03-08 14:16:54,400 5423 hyperkitty.views.mailman Archived message <01d801d95207$e34e52b0$a9eaf810$(a)sewagesludgeactionnetwork.com> to https://lists.ccalternatives.org/archives/list/ssan@lists.ccalternatives.or… INFO 2023-03-08 15:01:16,764 8917 hyperkitty.views.mailman Archived message <021201d95210$d2818c00$7784a400$(a)safecomputing.org> to https://lists.ccalternatives.org/archives/list/medialist@lists.ccalternativ… ERROR 2023-03-08 17:18:00,624 8915 django.security.DisallowedHost Invalid HTTP_HOST header: 'www.vsooth.com'. You may need to add 'www.vsooth.com' to ALLOWED_HOSTS. ERROR 2023-03-09 06:45:21,849 8916 django.security.DisallowedHost Invalid HTTP_HOST header: 'icook.tw'. You may need to add 'icook.tw' to ALLOWED_HOSTS. INFO 2023-03-09 07:46:33,940 8915 hyperkitty.lib.mailman Imported the new list redlatinleg(a)lists.ccalternatives.org from Mailman INFO 2023-03-09 08:01:35,590 8916 hyperkitty.views.mailman Archived message <02ab01d9529f$967afae0$c370f0a0$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat… INFO 2023-03-09 08:27:34,189 8919 hyperkitty.views.mailman Archived message <02c301d952a3$37279c70$a576d550$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat… INFO 2023-03-09 08:50:56,941 8916 hyperkitty.views.mailman Archived message <02cc01d952a6$7bdfbed0$739f3c70$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat… INFO 2023-03-09 13:03:42,385 8916 hyperkitty.views.mailman Archived message <032101d952c9$caf09990$60d1ccb0$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat… From: Zaheer Abbas <zaheer.psg(a)gmail.com> Sent: Thursday, March 9, 2023 9:00 AM To: csa(a)web-analysts.net; Christian <eric(a)safecomputing.org> Cc: mailman-users(a)mailman3.org Subject: Re: [MM3-users] new list not working Are you having issues with only this list or all lists? Did you configure and test smtp configuration? Check for mailman and smtp log files, usually placed in /var/log/mailman3/ On Thu, Mar 9, 2023 at 4:51 PM Christian via Mailman-users <mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org> > wrote: I created a new list using via the form. I sent a test message to the new list and it appears in the list archive. However I am not receiving the distribution. What should I check for please? Thank you ahead of time! _______________________________________________ Mailman-users mailing list -- mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org> To unsubscribe send an email to mailman-users-leave(a)mailman3.org <mailto:mailman-users-leave@mailman3.org> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message… This message sent to zaheer.psg(a)gmail.com <mailto:zaheer.psg@gmail.com> -- Zaheer Abbas Cell:+44-740-5511766 Personal Web: www.zaheerabbas.info <http://www.zaheerabbas.info> 'SAVE WATER ~ SAVE ENERGY ~ ~ SAVE EARTH

2 years

Re: REST API notes

by Barry Warsaw

On Jun 28, 2017, at 01:47 AM, tlhackque--- via Mailman-users wrote: >As a beginner with the REST API, I found it necessary to read code and >experiment to figure out how to get things done. Thanks for the feedback. Other than the work supporting Postorius and HyperKitty, I think you've done some of the more extensive exploration of the REST API, so your comments here (and greatly appreciated bug reports) are quite valuable, and I think will help us make the REST API both more usable and more discoverable. It's important to point out some history of the REST API. Originally, we used the restish library (which was Python 2 and eventually abandoned upstream). That choice informed the essentially dynamic nature of the REST API. Which means that unlike many other systems, we don't have -nor do I think we could if we wanted to- easily and automatically generate a static description of the REST API. When I switched to Falcon (which is both Python 3 compatible and well maintained upstream), I had to reimplement what I called "object based traversal". Falcon by default supports a routes based traversal mechanism, similar to Flask and other systems. But by plugging in (originally, by monkeypatching, but now through an official Falcon API) support for object based traversal, I was able to port the extensive existing REST implementation to the new underlying library fairly easily. All this to say that some of the problems you describe are due to historical accident. And that's served us well(-ish) while we only had to support mailmanclient and P/HK. But as the REST API is used in more places, those deficiencies are becoming more glaring. I think if we ever make progress on Lemme (our authenticating REST proxy), we'll likely encounter the same problems. But at this point, it's infeasible to completely reimplement the REST machinery. >There are a number of implementation choices that are unusual. No doubt. :) >There is some documentation in src/mailman/rest/docs. It's oriented toward >Python and intermixes the mailman client & REST API, apparently trying to >show equivalence. Much of that is because these are doctests, i.e. testable documentation. It's a tradeoff between making them useful as docs but also testable without too much clutter. It's very much oriented toward Python because that makes testability easier. It would be good to have documentation for pure HTTP/JSON consumers, but it would be imperative for that also to be testable so we're sure it remains accurate, aside from any requirements to keep the docs in sync. Suggestions and contributions would be very much welcome here. >The REST API has a hybrid interface: Requests are made with >application/x-www-form-urlencoded POST, PUT, PATCH and delete http requests. >Requests are also accepted as parameter strings. > >The responses are JSON. (This is rather surprising - one would expect JSON >requests - and I hope someday they'll be accepted, as the split complicates >clients. I suspect this evolved to simplify the initial Web GUI client >(Postorious), but it precludes using standard JSON-in/JSON-out client >libraries.) Absolutely. I want to support JSON-in/JSON-out. Again, the current mismatch is due to historical implementation decisions, but there's nothing in principle preventing us from accepting JSON in, afaict. We'd have to continue to accept both, for backward compatibility reasons. >Error responses are often misidentified as content-type: application/json, >but actually contain a text/plain error message. This isn't universally >true; for example the 401 response actually IS JSON. A client has to guess >and handle decoding exceptions. That's clearly a bug. The Content-Type should always be accurate. It would be a new feature to also support JSON error responses. >The API presents resources hierarchically, rooted at '/'. The top level >resources include /users, /lists. /addresses, /domains, /system > >The next level is a resource id. Kind of. It all depends on what the top-level resource is returned. This makes sense when you think about the object based traversal machinery. Each path component points internally to an object, and the subsequent path component is handled by that object. If that returns another object, then the next remaining path component is consumed by *that* object. And so on until we reach the end of the path. Then the object at the leaf responds to the HTTP command, and each object knows how to represent itself as JSON, and it knows its canonical location within the resource tree (there can be multiple paths to any particular object, but there's always one canonical location). >There is a an id which is stable for the lifetime of an object, and a current >name (which can be changed). For example, a list has a name like >mylist(a)example.com, and a list_id of mylist.example.com. But if you change >the list name, it becomes mynewname(a)example.net, while the list_id remains >mylist.example.com. Lists are weird in that they have two identifiers. One is the posting address, what we internally (and slightly incorrectly) call the "fully qualified list name". I don't particularly like that nomenclature anymore, but we live with it. A posting address can change, e.g. if you rename or rehome a list. The (RFC 2919) List-ID is assigned when the mailing list is created and it's immutable. Section 4 discourages changing the List-ID and Mailman takes that as a requirement. Rename a list or rehome it to a different domain on the same server and the List-ID will never change. Again for historical reasons, many APIs both internal and external used the posting address to identify the list, but this is wrong exactly because that can change. I've slowly been converting APIs to accept both the posting address and the List-ID when identifying the mailing list. New APIs generally accept only the List-ID. Bottom line, it's best to use the List-ID. >Lists are associated with "domains", which are the "domain part" of the >list's address. That is, the part after the @. This is sometimes referred >to as the "mail host", but there need not be a real host. Yep, again historical nomenclature. >A domain has to be created before you can create a list with an address in >that domain. Some APIs (e.g. `mailman create` CLI) will create the domain automatically if it doesn't already exist, and unless you disable that explicitly. >Resources are created with a POST to their top-level resource. To create a >domain, post to /domains with mail_host => the domain, and (optionally) >description => a description for the GUI. The response isn't JSON as one >would expect. In fact, it's an empty application/json response with a 201 >status. > >To create a list, one POSTs to /lists. This post takes a restricted set of >parameters; in the case of a list, just its fqdn_listname, (and an optional >style - which isn't well defined). The response isn't JSON as one would >expect. The Location header of the response contains a URL of the new list. What else would you expect? From my reading of books like RESTful Web Services (admittedly a long while ago), that's exactly the proper response to an appending POST. Return a 201 with a Location header to the new resource and empty content. >To configure the list, you have to follow up with a PUT or PATCH to that URL >+ /config This is where you can set the description, posting policy, etc. >It's unrealistic to do a PUT; even if you're cloning another list, there's no >programatic way to determine which attributes are writable. PATCH what you >know... Mailing list resources are somewhat unique in that they probably have the most properties of any resource/object in the system. That's not surprising if you think about it, but it does make PUTing to a mailing list more or less impractical. That's certainly not true of other, smaller resources though, and of course, you still want symmetry there (plus, implementation-wise, it's almost a no brainer to support both PUT and PATCH). >Members are associated with e-mail addresses - which belong to Users. Users can have multiple addresses, addresses can be linked to only one user, but may be unlinked, and members associate "subscribers" to mailing lists, where a subscriber is either a user-with-preferred-address, or an address. >You create a user by posting to /users with email => the email address, and >optionally display_name => the name string. (A user can also be implicitly >created by subscribing an e-mail address, but that gets confusing.) The >e-mail address is the primary email address for the user. More later. >Again, you get a Location header in the response, which you can use to PATCH >/preferences to set delivery_status, etc. .These preferences are part of a >hierarchy - many have a system default, a list default, the user default, and >a subscription to that list value. You can find the User associated with a >list by a GET of /addresses/address(a)example.net. Technically, the returned user isn't necessarily associated with a list. It *may* be subscribed to the list, but that relationship is represented by a member. >This GET returns two URLs: user => the user owning this address, and >self_link => the address object. *All* resources in the REST API have a self_link, and while that may seem redundant, it's not. As mentioned above, you can take various paths through the resource tree to get to a particular resource, but regardless of that, every resource has exactly one canonical location. That location is represented by the self_link. >Note also that there are both an email and original_email attribute. The >latter preserves case. The former is used internally by Mailman as a >resource key, etc. (Though exactly what happens if John(a)example.net and >john(a)example.net both subscribe is unclear.) They can't. Mailman is case-preserving case-insensitive for email addresses. Technically speaking, john(a)example.com and JOHN(a)example.com can be different mailboxes, but that never happens in practice anymore, and Mailman has always explicitly treated them as the same address. This goes back to the earliest days of Mailman. >Once a list is configured, you can add members. This requires the list_id - >which you don't (officially) have. So you do a GET on the list resource, to >get the list_id. Then you can subscribe a member with list_id => (the list >id), subscriber => email address. Optionally you can >pre_verify/confirm/approve the member and/or add a display_name.. POST to >/members. Again, you get a Location header back. You can't specify >everything you might like. as a creation attribute. So you may have to PATCH >the member to, for example, set the moderation_status. This is true, but it also kind of makes sense if you grok the way preferences work, i.e. hierarchically as you describe above. TBH, I don't particularly like the way preferences are modeled either internally or through the REST API, but I haven't been able to come up with anything better. >You may also need to PATCH the member /preferences if you want to set >list-level delivery status, etc. All true. In each of those cases, you may be talking to a different preference resource. >Consider adding at least an owner when creating a list. I think that would be a useful addition to the REST API for list creation. >One challenge is that almost everything requires multiple REST operations to >set up. But REST (by definition) is stateless. So the best you can do is >order operations & hope. I don't understand this. REST is stateless, but not all HTTP operations are idempotent. POST certainly isn't, by definition, so if you use it to create a new resource under a collection, you clearly cannot modify that resource through PUT or PATCH until the resource is created, which only happens when POST succeeds. >The mailman client examples refer to transactions (e.g in users.rst, there is >'transaction.commit(); - but REST can't hold state. It does appear that the >server uses DB transactions to ensure that any given REST operation is ACID, >but the composite operations (e.g. create a list and set it's config) can not >be Atomic. This is an architectural flaw in the API. I think what you're getting at is that some POSTs to create new resources do not accept the same parameters as the subsequent PUT or PATCH on the newly created resource. I think that may be true, but it's not universally so. E.g. POST on /domain creates a new domain, returning the location of the new resource. You can provide both a description and an owner, and you can also provide a mail_host. But mail_host is immutable, and PATCH or PUT support changing all mutable attributes on the domain. That should be the general rule; if an attribute on a resource is mutable, you should be able to PATCH and PUT it, *and* you should be able to specify all mutable attributes, plus some immutable ones on the POST that creates the resource. GET should return all attributes, mutable or immutable. This rule may not be strictly adhered to for all resources and collections, but I would consider that a bug, not an architectural flaw. AFAIK, this is all working as any well-defined web service should work. >Symbolic names (which are required) for attributes are in the >src/mailman/interfaces/(class).py. Please note that the interfaces under this package are for the *internal* API, which often, but not always, is exposed in the REST API. These two APIs serve different purposes so they cannot be a one-to-one mapping, and there are many resources in the REST API that don't correspond directly to internal objects, and there are many internal objects that are not exposed to the REST API. >What you can get/set comes from src/mailman/rest/(class).py. Yes, so clients of the REST API are served by objects in this package. Plugins and other internal operations are served by the objects that implement interfaces in the src/mailman/interfaces package. This is a strict separation of concerns, and it mirrors other 'external facing' interfaces, of which the command line is another example. Hope that helps. -Barry

7 years, 8 months

Mailman-users search results for query "Postorius"