August 2022 - Mailman-users - lists.mailman3.org

Emergency moderation and clearing out a lot of held messages
by bob B 20 Nov '22

20 Nov '22

We had an issue where a program went rogue and sent out 30k email alerts to one of our lists, so now I have a few questions. When the issue happened I turned on emergency moderation and we have resolved the issue so.. 1. How do I easily delete 30k of held messages? It looks like the GUI only allows me to do 200 at a time? is there a way to flush all held messages easily? 2. When I enabled Emergency moderation, I expected it to stop notifying the admin on every held message, but it appears that is not the case? Also, does it reply to the sender of every held message when Emergency moderation is on? it seems like the steps are: turn on "Emergency moderation" turn OFF "Notify users of held messages" turn OFF "Admin immed notify"

5 20

All addresses in a bounced list posting seem to be considered as bouncing
by Andy Smith 01 Oct '22

01 Oct '22

Hi, I noticed this happened (bounce.log): Jul 09 12:46:51 2022 (71506) Member xxxxx(a)xxxxx.force9.co.uk already scored a bounce on list users.mailman.bitfolk.com today. Jul 09 12:46:51 2022 (71506) Bounce message for a non subscriber: Email users-leave(a)mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com Jul 09 12:46:51 2022 (71506) Member andy(a)bitfolk.com on list users.mailman.bitfolk.com, bounce score = 1. Jul 09 12:46:51 2022 (71506) Bounce message for a non subscriber: Email users(a)mailman.bitfolk.com is not a subcriber of users.mailman.bitfolk.com What happened here is that a reply to one of my posts was rejected by subscriber xxxxx(a)xxxxx.force9.co.uk at smtp time (Exim mainlog): 2022-07-09 12:46:13 1oA9qc-007qOx-0T ** xxxxx(a)xxxxx.force9.co.uk R=dnslookup T=remote_smtp H=mx.avasin.plus.net [212.159.8.200] X=TLS1.2:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256 CV=yes DN="businessCategory=Private Organization,jurisdictionOfIncorporationCountryName=GB,serialNumber=03279013,C=GB,L=Sheffield,O=Plusnet plc,CN=mx.avasin.plus.net": SMTP error from remote mail server after end of data: 552 Spam Message Rejected The message contained my email address (andy(a)bitfolk.com) in quoted text, since it was a reply to a message of mine, and also contained addresses from the list's footer, which accounts for the log messages about users-leave(a)mailman.bitfolk.com and users(a)mailman.bitfolk.com. I'm concerned that Mailman seems to have scored my address as bouncing even though it isn't, merely because it appears inside a message that was rejected. Is that expected? Is there any way to avoid it? I see that Mailman has a bunch of heuristics to parse bounce messages. In this case though, there shouldn't have been a bounce message received to users-bounces@, as this was an SMTP-time rejection. Mailman seems to have parsed addresses out of the actual mail it was sending, rather than only scoring the subscriber address it was trying to send a list posting to. Cheers, Andy

3 11

invalid parameter response of the rest-api
by Jacob Sievert 22 Sep '22

22 Sep '22

Hello, Since a few days we get an error response from our automated membership script. We use the following code to control the membership for some lists: ``` def mailinglist_manage(list_address, members): """ make sure all 'members' are present on the list and no remove everyone else list_address : mail address of mailing list to manage addresses : mail address that are supposed to be on the list """ print("[*] update subscriptions for {}".format(list_address)) ml = mmc.get_list(list_address) # make sure we dont spam users ml.settings['send_welcome_message'] = False ml.settings.save() addresses = set(members.keys()) ml_members = ml.members ml_addresses = set((x.address.__str__() for x in ml_members)) for addr in ml_addresses - addresses: print("unsubscribe {} from {}".format(addr, list_address)) ml.unsubscribe(addr) for addr in addresses - ml_addresses: print("subscribe {} to {}".format(addr, list_address)) ml.subscribe(addr, members[addr], pre_verified=True, pre_confirmed=True, pre_approved=True) ``` Since Tuesday without any update performed by us we get the following error from the script when it's running: ``` Traceback (most recent call last): File "./mail_user_control.py", line 423, in <module> mailinglist_manage("XXX(a)EXAMPLE.COM", members) File "./mail_user_control.py", line 412, in mailinglist_manage pre_verified=True, pre_confirmed=True, pre_approved=True) File "/root/.local/lib/python3.6/site-packages/mailmanclient/restobjects/mailinglist.py", line 339, in subscribe response, content = self._connection.call('members', data) File "/root/.local/lib/python3.6/site-packages/mailmanclient/restbase/connection.py", line 112, in call error_msg, response, None) urllib.error.HTTPError: HTTP Error 400: Invalid Parameter "subscriber": . ``` If you have an idea why that code now returns this. I would be thankful. regards Jacob Sievert

4 7

Re: mail stuck in "out" queue
by Dmitry Makovey 16 Sep '22

16 Sep '22

So our mailman got stuck again, but I did follow your advise. Here's what I've found: On 3/29/19 2:58 PM, Mark Sapiro wrote: > On 3/29/19 2:28 PM, Dmitry Makovey wrote: >> I've had encountered this situation now twice: mail is being stuck in >> "out" queue and nothing in the logs indicates the issue. Is there a way >> to ask mailman-core politely to attempt delivery (perhaps with increased >> debugging in the logs) *without* restarting the core (which does solve >> *this* problem but creates other problems, like availability of the >> mailman as a whole) > > > Is the "out" runner running? yes: # docker top mailman-core | grep -F out: 100 21681 13215 0 Mar28 ? 01:14:16 /usr/local/bin/python /usr/local/bin/runner --runner=out:0:1 -C /etc/mailman.cfg > ps -fwwA |grep runner=out > > If not, you should be able to start it as the Mailman user with a > command like > > /path/to/python3 /path/to/mailman/bin/runner -C /path/to/mailman.cfg > --runner=out:0:1 > > See 'ps -fwwA|grep runner' for examples > > If the out runner is running, is there a file named *.bak along with all > the *.pck files in queue/out/ there is one, yes: # ls -l /srv/mail/mailman/core/var/queue/out/*.bak -rw-rw---- 1 100 65533 21512 Apr 14 19:45 /srv/mail/mailman/core/var/queue/out/1555296331.8386078+2c905825b82708e5a682b872b0aff320c9dd0b2e.bak what does it indicate and what should my steps be to resolve this (other than restarting queues)? > And if it's not, is there anything in Mailman's logs to indicate why it > died? > > Also, 'mailman restart' should only take a matter of seconds, but if the > out runner is running but not processing its queue, I think sending it a > SIGINT should restart it. SIGINT killed the process allright, but it didn't auto-respawn. I ended up spawning it with: /usr/local/bin/python /usr/local/bin/runner --runner=out:0:1 -C /etc/mailman.cfg but soon realized it doesn't go to background nicely so I've nohup'ed it in the end. but it did solve my problem. -- Sr System and DevOps Engineer SoM IRT

5 12

Restore/export existing Mailman lists + hyperkitty archives on a new server
by Kimmo L 16 Sep '22

16 Sep '22

Hello! My plan is to install fresh Mailman3 host, which means that i need to migrate/transfer my existing mailman mailing lists + hyperkitty archives to new host. Unfortunately i didnt find any good documentation about this. I tried myself to export and import mailmanweb/mailman databases (from old PostgreSQL to new PostgreSQL) and recreate indexes(mailman-web update_index). Now i can see and search hyperkitty archives, but if try to open one of the mailing list, i get error - HTTP Error 500: {"title": "500 Internal Server Error"} So im pretty sure, that I'm gonna have to transport some more stuff over, which I have no idea. Maybe existing users and user rights. If its not a big trouble, can someone kindly share, what exactly should i need to export/import and what would be the right steps/commands/moves. Br, Kimmo

3 50

Message-Footer after migrating to Mailman3
by christopher.claus＠tgcamberg1848.de 13 Sep '22

13 Sep '22

Hello together, I migrated a number of lists from mailman2 to mailman3 using the import functionality. Everthing works fine and the new lists are in use. Currently I´m using the maxking-docker-images with postorius: Mailman Core Version GNU Mailman 3.3.3 (Tom Sawyer) Mailman Core API Version 3.1 Mailman Core Python Version 3.8.5 (default, Feb 19 2021, 09:27:16) [GCC 9.3.0] After this migration I started a refactoring with the list-styles specially with message-footers. Already the old lists in mailman2 were confused in some configuration details an I was clear to import this "misconfiguration" and clean it up afterwards. But this was much more difficult than expected: some lists are using a footer (from the mailman2-config) and there is no record in the template-section visible. Further I found no configuration item which points to this "old" footer. I found the file [code] /usr/lib/python3.8/site-packages/mailman/templates/de/list:member:generic:footer.txt [/code] which includes the old footer: [code] _______________________________________________ $display_name Mailingliste -- $listname Zur Abmeldung von dieser Mailingliste senden Sie eine Nachricht an ${short_listname}-leave@${domain} [/code] but no configuration item to disable this behaviour. I created a new template list:member:regular:footer for the list to override the footer. But the old footer is still used - in detail: the resulting message sent to all subcribers of the list contains only the footer and the content sent to the list is an attachement of that "footer-message". This is the basic reason why I want to change it. In the logfiles I found an error and a warning: [code] ERROR 2021-05-24 16:22:22,895 1648 django.security.DisallowedHost Invalid HTTP_HOST header: 'tgc_mailman_web:8000'. The domain name provided is not valid according to RFC 1034/1035. WARNING 2021-05-24 16:22:22,896 1648 django.request Bad Request: /postorius/api/templates/list/homepage.gruppe.tgcamberg1848.de/list:member:regular:footer [/code] Obviously I have a problem with the hostname. I set [code] POSTORIUS_TEMPLATE_BASE_URL=http://tgc_mailman_web:8000 [/code] where tgc_mailman_web is my container name. This URL ist stored in the DB-table "template" for the name=list:member:regular:footer. A call to my fqdn or to localhost [code] wget http://localhost:8000/postorius/api/templates/list/homepage.gruppe.tgcamber… [/code] will return the footer itself - this works fine. Therefore, should i set POSTORIUS_TEMPLATE_BASE_URL to http://localhost:8000? I think, resolving this issue will use the newly configured footer. But I started with the intention to remove alle footers due to the attachement situation described above. Does anybody have an idea how to change the configuration of my list? Are these problems probably related to the import? Currently we are using more then 25 lists with a lot of users. Manually recreating all lists is not possible an could only be a test for a single list. Best regards, chrclaus

6 18

mailman3 not working for mailman21 lists
by blup＠fortytwo.ch 06 Sep '22

06 Sep '22

I installed mailman3 on a server where mailman21 was installed before. Newly created mailinglists work fine. But if I create mailinglists with a name that was previously used on mailman21 nothing works (same behaviour with and without import of old configs and archives; no logs). I purged all old mailman21 config and data and also old postfix aliases (or at least I do think so). If I send a mail to old list list1(a)a.b and new list list2(a)a.b I get the following logs /var/log/mail.log Aug 21 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=<list2(a)c.a.b>, orig_to=<list2(a)a.b>, relay=127.0.0.1[127.0.0.1]:8024, delay=0.31, delays=0.17/0.01/0/0.13, dsn=2.0.0, status=sent (250 Ok) Aug 21 22:17:40 basilisgg postfix/lmtp[1728]: ED2171120BF: to=<list1(a)c.a.b>, orig_to=<list1(a)a.b>, relay=127.0.0.1[127.0.0.1]:8024, delay=0.31, delays=0.17/0.01/0/0.13, dsn=2.0.0, status=sent (250 Ok) /var/log/mailman3/smtp.log ug 21 22:17:40 2022 (934) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Aug 21 22:17:40 2022 (934) Peer: ('127.0.0.1', 44746) Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) handling connection Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'LHLO c.a.b' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'MAIL FROM:<....>' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) sender: .... Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:<list2(a)c.a.b>' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) recip: list2(a)c.a.b Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'RCPT TO:<list1(a)c.a.b>' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) recip: list1(a)c.a.b Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'DATA' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) Data: b'QUIT' Aug 21 22:17:40 2022 (934) ('127.0.0.1', 44746) connection lost Aug 21 22:17:40 2022 (934) Connection lost during _handle_client() Aug 21 22:17:46 2022 (936) <166111306097.933.1775779871266718007@...> smtp to list2(a)a.b for 1 recips, completed in 3.1437485218048096 seconds How can I get more info on why the mail for list1 isn't processed? Where could old mailman21 or postfix info on list1 be stored that causes mailman3 to not handle the list properly? How can I diff the config between list1 and list2 from the command line? Regards Diana

5 12

FYI: Django security releases issued: 4.0.6 and 3.2.14
by Guillermo Hernandez (Oldno7) 29 Aug '22

29 Aug '22

https://www.djangoproject.com/weblog/2022/jul/04/security-releases/ Django security releases issued: 4.0.6 and 3.2.14 Posted by *Mariusz Felisiak* on Julio 4, 2022 In accordance with our security release policy <https://docs.djangoproject.com/en/dev/internals/security/>, the Django team is issuing Django 4.0.6 <https://docs.djangoproject.com/en/dev/releases/4.0.6/> and Django 3.2.14 <https://docs.djangoproject.com/en/dev/releases/3.2.14/>. These release addresses the security issue detailed below. We encourage all users of Django to upgrade as soon as possible. CVE-2022-34265: Potential SQL injection via Trunc(kind) and Extract(lookup_name) arguments Trunc() and Extract() database functions were subject to SQL injection if untrusted data was used as a kind/lookup_name value. Applications that constrain the lookup name and kind choice to a known safe list are unaffected. This security release mitigates the issue, but we have identified improvements to the Database API methods related to date extract and truncate that would be beneficial to add to Django 4.1 before it's final release. This will impact 3rd party database backends using Django 4.1 release candidate 1 or newer, until they are able to update to the API changes. We apologize for the inconvenience. Thanks Takuto Yoshikai (Aeye Security Lab) for the report. This issue has severity "high" according to the Django security policy. -- ___________________________________________ Mailman's content filtering has removed the following MIME parts from this message. Content-Type: image/png Name: firma-GHP-emails.png Replaced multipart/alternative part with first alternative.

5 18

Migrated from MM2 to MM3, all is well except *one* list
by Scott Courtney 28 Aug '22

28 Aug '22

Hello, all I have migrated a multi-list server from MM2 to MM3, on Debian 11, and migrated about half a dozen lists with fairly large archives. Absolutely everything has been working for a couple of months, but I am still unable to get one specific list to send mail to its subscribers. Of course, the first place I started was the logs, but that's the problem -- even with logging turned up, it appears to be failing silently as it connects to the MTA. Here's a summary of the environment: OS: Debian Linux 11, 64-bit, on a virtual machine MTA: Exim 4.94.2 List server: Mailman 3.3.3 (installed from Debian repo) Database: MariaDB 10.5.15 All daemons are running on the same host, and communication is either UNIX socket or loopback network interface, for security. Web server: Apache 2.4.53, with proxy to WSGI per the MM3 setup guide Symptom: There are six lists on the server. All six can accept inbound traffic and will post the messages to the (always private, in this environment) archives. The web interface is working perfectly. Five of the lists also can send out messages to subscribers, but the sixth list cannot send traffic to the MTA, or is not attempting to do so. Tests run so far (all resulting in successful posting to archives, but no outbound mail): * Originate mail from inbound SMTP to the list address. * Originate mail from the web UI as a member/moderator. * Originate mail from the web UI as the site admin (also the list owner). * Originate mail from both inbound SMTP and web UI by multiple users. Log data for the inbound traffic (typical test message, some data redacted): 2022-08-27 15:11:57 1oS1Dn-003AWH-Hz <= REDACTED H=REDACTED:25 P=esmtps X=TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_128_GCM:128 CV=no K S=2552 DKIM=REDACTED id=REDACTED T="TEST 8.27.001" 2022-08-27 15:11:57 1oS1Dn-003AWH-Hz => MY_LIST(a)lists.MY_DOMAIN R=mailman_main_director T=mailman3_transport H=127.0.0.1 [127.0.0.1] I=[127.0.0.1] C="250 Ok" 2022-08-27 15:11:57 1oS1Dn-003AWH-Hz Completed Log data for MM3 smtp.log (again, partially redacted): Aug 27 15:11:57 2022 (690182) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin) Aug 27 15:11:57 2022 (690182) Peer: ('127.0.0.1', 48170) Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) handling connection Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'LHLO REDACTED' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'MAIL FROM:<REDACTED>' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) sender: REDACTED Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'RCPT TO:<erevnite(a)lists.4th.com>' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) recip: MY_LIST(a)lists.MY_DOMAIN Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'DATA' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) Data: b'QUIT' Aug 27 15:11:57 2022 (690182) ('127.0.0.1', 48170) connection lost Aug 27 15:11:57 2022 (690182) Connection lost during _handle_client() Originally I thought the "connection lost" indicated a failure, but that is happening for all the lists, and also on successful traffic. Since it occurs after the QUIT command in the SMTP session, and based on some online searching, my current understanding is that this is harmless and simply indicates the MTA closed the socket after the QUIT. (Please correct me if that is wrong.) Log data for mailman.log, with partial redaction: Aug 27 15:11:57 2022 (690181) ACCEPT: <REDACTED> Aug 27 15:11:59 2022 (690178) HyperKitty archived message <REDACTED> to https://lists.MY_DOMAIN/hyperkitty/list/MY_LIST@lists.MY_DOMAIN/message/G3D… 19:11:59 [Q] INFO Enqueued 518 19:11:59 [Q] INFO Enqueued 519 19:11:59 [Q] INFO Enqueued 520 19:11:59 [Q] INFO Enqueued 521 19:11:59 [Q] INFO Enqueued 522 19:11:59 [Q] INFO Enqueued 523 19:11:59 [Q] INFO Enqueued 524 INFO 2022-08-27 19:11:59,648 690164 hyperkitty.views.mailman Archived message <REDACTED> to https://lists.MY_DOMAIN/hyperkitty/list/MY_LIST@lists.MY_DOMAIN/message/G3D… [pid: 690164|app: 0|req: 4688541/4688541] 2600:3c04::f03c:91ff:fe02:9cc1 () {64 vars in 1255 bytes} [Sat Aug 27 19:11:59 2022] POST /hyperkitty/api/mailman/archive?key=xombsjgLDqe6d1oxvBaQWH4ZT81uCEjH => generated 113 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 160 bytes (1 switches on core 0) 19:12:00 [Q] INFO Process-1:4 processing [update_from_mailman] 19:12:00 [Q] INFO Process-1:3 processing [sender_mailman_id] 19:12:00 [Q] INFO Process-1:2 processing [rebuild_thread_cache_new_email] 19:12:00 [Q] INFO Process-1:1 processing [compute_thread_positions] 19:12:00 [Q] INFO Processed [rebuild_thread_cache_new_email] 19:12:00 [Q] INFO Process-1:2 processing [rebuild_mailinglist_cache_recent] 19:12:00 [Q] INFO Processed [sender_mailman_id] 19:12:00 [Q] INFO Process-1:3 processing [rebuild_mailinglist_cache_for_month] 19:12:00 [Q] INFO Process-1:1 processing [check_orphans] 19:12:00 [Q] INFO Processed [compute_thread_positions] 19:12:00 [Q] INFO Processed [rebuild_mailinglist_cache_for_month] 19:12:00 [Q] INFO Processed [check_orphans] 19:12:00 [Q] INFO Processed [rebuild_mailinglist_cache_recent] 19:12:00 [Q] INFO Processed [update_from_mailman] [pid: 690164|app: 0|req: 4693691/4693691] 184.94.240.88 () {60 vars in 929 bytes} [Sat Aug 27 19:21:50 2022] HEAD / => generated 0 bytes in 2 msecs (HTTP/1.1 301) 6 headers in 204 bytes (1 switches on core 0) [pid: 690164|app: 0|req: 4693694/4693694] 184.94.240.88 () {60 vars in 1009 bytes} [Sat Aug 27 19:21:50 2022] HEAD /postorius/lists/ => generated 7136 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 163 bytes (1 switches on core 1) There is nothing at all in the Exim log indicating an outbound transaction was attempted by Mailman. I know Exim and Mailman are communicating because all the lists use the same SMTP configuration for Exim and Mailman (i.e., the non-working list is not in any way exceptional). I've done a lot of searching online to find a solution, and I did see that a couple of others had asked about this type of problem, but the only one I found that got a response had an issue that didn't apply in my small environment. Absent any other ideas, I'm going to try inserting some trace logging into the Python code, but I can't figure out what's wrong here. I've even gone to the point of looking at the SQL rows to verify this list is exactly parallel to another working list, except its name and integer ID numbers for various unique auto-increment keys. Any suggestions are appreciated. Thanks!

2 2

Using REST with CLI using curl .... help needed
by Krešimir Mihalj 27 Aug '22

27 Aug '22

Hi ! I need simple add/delete member to list using rest ..... but all documentation is for python and not helpfull for me, .... and I need these in CLI using curl command .... To add member using CLI i tried this command: curl -d "{'list_id' : 'list.domain.com', 'subscriber' : 'user(a)email.com'}" -H 'Content-Type: application/json' --user restadmin:restpass http://localhost:8001/3.0/members and I got this response: Missing parameters: list_id, subscriber What is wrong in above command here ? ..... I need to add user: user(a)email.com to list: list(a)domain.com Also, if someone is willing to give example: delete user: user(a)email.com from list: list(a)domain.com I actually create PHP application .... but CLI example is easy to reproduce in PHP function AddUserToList($user,$lista) { $url = 'http://llocalhost:8001/3.0'; $request_url = $url . '/members'; $data = array("list_id" => $lista,"subscriber" => $user ); $postdata = json_encode($data); $ch = curl_init($request_url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_USERPWD, "restadmin:restpass"); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json')); $result = curl_exec($ch); curl_close($ch); print_r ($result); } AddUserToList('user(a)email.com','list.domain.com');

6 8

2022

2021

2020

2019

2018

2017

2016

Mailman-users August 2022