February 2022 - Mailman-users - lists.mailman3.org

Mass un/subscribe optimization?
by Bryan Loniewski Feb. 28, 2022

Feb. 28, 2022

Hi -- We've noticed slowness in mass un/subscribing users; looking through the issue(s) log we see both #419 [1] and #627 [2] open and related. From the cmd line we get better performance, but with the subscriber size on many of our Faculty/Staff/Students ad-hoc lists (think 1k-15K subscribed addresses) and with our number of lists (2600) and unique owners utilizing these actions it would be helpful to optimize the mass actions. We wanted to get some thoughts and guidance on what needs to be optimized/redesigned/fixed? Or is there already something in the works, but needs more time? We may be able to provide resources to encourage and/or assist with any development and/or testing... We run a split core/web setup with Apache,mod_wsgi and MySQL backend (AWS hosted). [1] - https://gitlab.com/mailman/mailman/-/issues/419 [2] - https://gitlab.com/mailman/mailman/-/issues/627

2 1

Migration of archives from Sympa to MM3
by Tom Fillmore Feb. 28, 2022

Feb. 28, 2022

Hi, all - At the moment I have a site that uses Sympa. It has more than 150 lists. Moving existing lists to MM3 looks doable the archives are a different story. Has anyone ever migrated archives from Sympa to MM3? Sorry if this is answered somewhere else... Thanks T Fillmore southern California The earth is but one country and mankind its citizens... - Baha'u'llah

2 2

Migrating archives from mailman2 to mailman3 with attachments
by bob B Feb. 28, 2022

Feb. 28, 2022

So I see the process for migrating an archive from mailman2 to mailman3 https://docs.mailman3.org/en/latest/migration.html.................. i.e python manage.py hyperkitty_import -l foo-list(a)example.com $var_prefix/archives/private/foo-list.mbox/foo-list.mbox but does the migration handle attachments?? attachments are in ..../archives/private/foo-list/attachments. I assume this is if you are upgrading in place? I am migrating to a new server and my plan is to follow these steps but copy the config.pck. and .mbox files to the new server and import them, but then it looks like I need to copy the attachment directory? if so how do I import the attachments to mailman3?

2 1

sending messages in response to posts from non-members
by David Bremner Feb. 27, 2022

Feb. 27, 2022

I'd like to turn off the "bounce messages" sent by mailman to non-members. My reasoning is that most such posts are spammers, and often use forged return-paths / From's, so I end of sending steady stream of backscatter to amazon.com etc... Is this configurable (without postorius) somehow? d

2 2

"autorespond to list owner" setting when importing from mailman2 to mailman3
by bob B Feb. 25, 2022

Feb. 25, 2022

In mailman2 a list was moderated, the admin would also get an email to approve the message, our lists mainly have admins and no moderators. The admins did the moderation. In mailman3 it looks this is now a setting "autorespond to list owner" was this a setting in mailman 2? Anyway, in importing email list from mailman 2 to 3, it appears that this setting is set to "No automatic response", is that right? So does this means if I import all my lists from mailman 2, then the admins will not get moderation notifications until I make that setting change on each list?

2 4

Python 3.9 support
by dancab＠caltech.edu Feb. 25, 2022

Feb. 25, 2022

I just upgraded our Mailman3 system to use Python 3.9. About 6 hours after the deployment there was a PyMySQL error and delivery of messages stopped until I restarted the services. I'm pasting the full stack trace error from mailman.log below. It mentions PyMySQL. We have PyMySQL-1.0.2 installed. Thanks for your help. ---- Dan Caballero Systems Administrator Academic Computing Solutions IMSS - Caltech https://imss.caltech.edu Feb 23 23:27:10 2022 (300) Uncaught runner exception: (pymysql.err.OperationalError) (1053, 'Server shutdown in progress') [SQL: SELECT anon_1.member_id AS anon_1_member_id, anon_1.member__member_id AS anon_1_member__member_id, anon_1.member_role AS anon_1_member_role, anon_1.member_list_id AS anon_1_member_list_id, anon_1.member_moderation_action AS anon_1_member_moderation_action, anon_1.member_address_id AS anon_1_member_address_id, anon_1.member_preferences_id AS anon_1_member_preferences_id, anon_1.member_user_id AS anon_1_member_user_id, anon_1.member_bounce_score AS anon_1_member_bounce_score, anon_1.member_last_bounce_received AS anon_1_member_last_bounce_received, anon_1.member_last_warning_sent AS anon_1_member_last_warning_sent, anon_1.member_total_warnings_sent AS anon_1_member_total_warnings_sent FROM (SELECT `member`.id AS member_id, `member`._member_id AS member__member_id, `member`.`role` AS member_role, `member`.list_id AS member_list_id, `member`.moderation_action AS member_moderation_action, `member`.address_id AS member_address_id, `member`.preferences_id AS member_preferences_id, `member`.user_id AS member_user_id, `member`.bounce_score AS member_bounce_score, `member`.last_bounce_received AS member_last_bounce_received, `member`.last_warning_sent AS member_last_warning_sent, `member`.total_warnings_sent AS member_total_warnings_sent FROM `member`, address WHERE `member`.list_id = %(list_id_1)s AND `member`.`role` = %(role_1)s AND address.email = %(email_1)s AND `member`.address_id = address.id UNION SELECT `member`.id AS member_id, `member`._member_id AS member__member_id, `member`.`role` AS member_role, `member`.list_id AS member_list_id, `member`.moderation_action AS member_moderation_action, `member`.address_id AS member_address_id, `member`.preferences_id AS member_preferences_id, `member`.user_id AS member_user_id, `member`.bounce_score AS member_bounce_score, `member`.last_bounce_received AS member_last_bounce_received, `member`.last_warning_sent AS member_last_warning_sent, `member`.total_warnings_sent AS member_total_warnings_sent FROM `member`, address, user WHERE `member`.list_id = %(list_id_2)s AND `member`.`role` = %(role_2)s AND address.email = %(email_2)s AND `member`.user_id = user.id AND user._preferred_address_id = address.id) AS anon_1] [parameters: {'list_id_1': 'avery-disc.caltech.edu', 'role_1': 1, 'email_1': 'nandrade(a)caltech.edu', 'list_id_2': 'avery-disc.caltech.edu', 'role_2': 1, 'email_2': 'nandrade(a)caltech.edu'}] (Background on this error at: http://sqlalche.me/e/13/e3q8) Feb 23 23:27:10 2022 (300) Traceback (most recent call last): File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) pymysql.err.OperationalError: (1053, 'Server shutdown in progress') The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/mailmanve/lib/python3.9/site-packages/mailman/core/runner.py", line 176, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/core/runner.py", line 269, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/runners/outgoing.py", line 111, in _dispose self._func(mlist, msg, msgdata) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/mta/deliver.py", line 86, in deliver refused = agent.deliver(mlist, msg, msgdata) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/mta/base.py", line 164, in deliver member = mlist.members.get_member(recipient) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/model/roster.py", line 120, in get_member memberships = self._get_all_memberships(email) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/model/roster.py", line 116, in _get_all_memberships return members_a.union(members_u).all() File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3373, in all return list(self) File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__ return self._execute_and_instances(context) File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement ret = self._execute_context( File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context self._handle_dbapi_exception( File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception util.raise_( File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context self.dialect.do_execute( File "/opt/mailmanve/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/opt/mailmanve/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1053, 'Server shutdown in progress') [SQL: SELECT anon_1.member_id AS anon_1_member_id, anon_1.member__member_id AS anon_1_member__member_id, anon_1.member_role AS anon_1_member_role, anon_1.member_list_id AS anon_1_member_list_id, anon_1.member_moderation_action AS anon_1_member_moderation_action, anon_1.member_address_id AS anon_1_member_address_id, anon_1.member_preferences_id AS anon_1_member_preferences_id, anon_1.member_user_id AS anon_1_member_user_id, anon_1.member_bounce_score AS anon_1_member_bounce_score, anon_1.member_last_bounce_received AS anon_1_member_last_bounce_received, anon_1.member_last_warning_sent AS anon_1_member_last_warning_sent, anon_1.member_total_warnings_sent AS anon_1_member_total_warnings_sent FROM (SELECT `member`.id AS member_id, `member`._member_id AS member__member_id, `member`.`role` AS member_role, `member`.list_id AS member_list_id, `member`.moderation_action AS member_moderation_action, `member`.address_id AS member_address_id, `member`.preferences_id AS member_preferences_id, `member`.user_id AS member_user_id, `member`.bounce_score AS member_bounce_score, `member`.last_bounce_received AS member_last_bounce_received, `member`.last_warning_sent AS member_last_warning_sent, `member`.total_warnings_sent AS member_total_warnings_sent FROM `member`, address WHERE `member`.list_id = %(list_id_1)s AND `member`.`role` = %(role_1)s AND address.email = %(email_1)s AND `member`.address_id = address.id UNION SELECT `member`.id AS member_id, `member`._member_id AS member__member_id, `member`.`role` AS member_role, `member`.list_id AS member_list_id, `member`.moderation_action AS member_moderation_action, `member`.address_id AS member_address_id, `member`.preferences_id AS member_preferences_id, `member`.user_id AS member_user_id, `member`.bounce_score AS member_bounce_score, `member`.last_bounce_received AS member_last_bounce_received, `member`.last_warning_sent AS member_last_warning_sent, `member`.total_warnings_sent AS member_total_warnings_sent FROM `member`, address, user WHERE `member`.list_id = %(list_id_2)s AND `member`.`role` = %(role_2)s AND address.email = %(email_2)s AND `member`.user_id = user.id AND user._preferred_address_id = address.id) AS anon_1] [parameters: {'list_id_1': 'avery-disc.caltech.edu', 'role_1': 1, 'email_1': 'nandrade(a)caltech.edu', 'list_id_2': 'avery-disc.caltech.edu', 'role_2': 1, 'email_2': 'nandrade(a)caltech.edu'}] (Background on this error at: http://sqlalche.me/e/13/e3q8)

2 4

Hyperkitty failure?
by Stephen Daniel Feb. 24, 2022

Feb. 24, 2022

Mailman3 people -- Hyperkitty is failing in my new setup. In the log I see messages that look like this: Feb 24 11:43:44 2022 (28621) HyperKitty failure on https://lists.example.com/archives/api/mailman/archive: <html> <head><title>401 Authorization Required</title> /opt/mailman/mm/mailman-hyperkitty.cfg looks like this: [general] base_url: https://lists.example.com/archives api_key: SOMESTRING settings_local.py includes this line: MAILMAN_ARCHIVER_KEY = 'SOMESTRING' and mailman.cfg contains [archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /opt/mailman/mm/mailman-hyperkitty.cfg Any suggestions on how to get hyperkitty working? Thanks! -- Steve

2 2

cleanup lost users completly - mailman shell
by Paul Paulchen Feb. 21, 2022

Feb. 21, 2022

Hi community, I was wondering if there is a walk-through for the following issue - or is it even an issue? We got plenty lists with a lot of users come and go from time to time. Thanks to the thread, I understood how to pick on adress and delete them via the mailman shell; https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/… >>> um = getUtility(IUserManager) >>> usr = um.get_user('user(a)example.com') >>> um.delete_user(usr) >>> commit() Thanks to this thread, I know how to find the lost souls in our mailman environment https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/… >>> um = getUtility(IUserManager) >>> for usr in um.users: ... if len(list(usr.memberships.members)) > 0: ... continue ... adr = usr.addresses[0] ... print(f'Name: {adr.display_name}, Email: {adr.email}, Created: {usr.created_on}') So I can see them now. Is there a technique, to clean them up? To get rid of them out of the system? I am not a python programmer, but I guess there is a way to use, instead of a single user, a list of users in the delete command of the mailman shell? e.g. (sorry if this example leads to any pain for any coder/ programmer) user1(a)example.com user2(a)example.com user3(a)example.com >>> um = getUtility(IUserManager) >>> usr = um.get_user('user1(a)example.com','user2(a)example.com','user3(a)example.com') >>> um.delete_user(usr) >>> commit() Hope I made my intention clear. I want to get rid of the users who don't belong/ or are subscribe to a list anymore. Thanks in advance! Paul

4 3

aborted connection
by Sandor Dibuz Feb. 20, 2022

Feb. 20, 2022

Am I right expecting that such warnings are harmless and probably come from timeouts and/or unterminated connections to the DB? Feb 19 14:15:18 localhost mariadbd[28859]: 2022-02-19 14:15:18 22154 [Warning] Aborted connection 22154 to db: 'mailman3' user: 'mailman3' host: 'localhost' (Got an error reading communication packets) Thank you. Sanyi

2 4

I did something stupid. Can i back out of it?
by William Oliver Feb. 20, 2022

Feb. 20, 2022

So... I run a small mailinglist using mailman3 that has been workiing great for a couple of months. Last week, all my users from gmail got kicked off because gmail decided that the list messages were "unsolicited email". I didn't find out about it until the users hit their mailman3 bounce limit. I *thought* would be seeing all the bounce messages, but apparently not. I didn't see any config settings for "see all the bounce messages" so i thought to myself, "Self, well maybe that's because my mailinglist username doesn't have a mailbox in /home/listname". So, I decided to try that. I created /home/listname/Maildir/* and logged in as fplist to see if it would put mail there. In order to log in as listname, I gave the user listname a password. Then the world ended. The mailinglist works OK for *new* posts, but it is now impossible to *reply* to posts. So, if I post as a member from user(a)somewhere.org to listname(a)example.com, it works. If I hit reply, with the From: line rewritten as listname(a)example.com, it fails. The error I get is: The reported error was “Error performing TLS handshake: An unexpected TLS packet was received.”. My suspicion is that there are two passwords floating around for "listname" -- one in mailman3 and the one in /etc/shadow, but I don't know. In retrospect, this was not my finest moment. But is there an easy way to back out of this? Thanks! billo

2 8