- Mailman-users - lists.mailman3.org

Memory usage
by Adi Kriegisch 13 Sep '18

13 Sep '18

Dear all, I've been using mailman2 for quite some time and I am considering upgrading to mailman3. In a test setup I noticed that mailman3 with just a basic setup (Debian with mailman3 package and sqlite as backend using exim4) uses over 700MB RAM (RSS): | USER VSZ RSS COMMAND | list 73072 54568 /usr/bin/python3 /usr/lib/mailman3/bin/master | list 73100 54500 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=archive:0:1 | list 73100 54572 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=bounces:0:1 | list 73100 54560 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=command:0:1 | list 73100 54600 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=in:0:1 | list 146832 54700 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=lmtp:0:1 | list 73100 54808 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=nntp:0:1 | list 73100 54652 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=out:0:1 | list 73100 54504 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=pipeline:0:1 | list 150516 58228 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=rest:0:1 | list 73100 54508 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=retry:0:1 | list 73100 54500 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=virgin:0:1 | list 73100 54608 /usr/bin/python3 /usr/lib/mailman3/bin/runner --runner=digest:0:1 This is with only one test list configured that just has one member. Compared to a mailman2 setup with many lists with up to 300 members that only uses around 70MB RAM: | USER VSZ RSS COMMAND | list 63724 1396 /usr/bin/python /usr/lib/mailman/bin/mailmanctl -q start | list 62896 10660 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s | list 61156 1792 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s | list 58148 1808 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=CommandRunner:0:1 -s | list 73116 23316 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s | list 55464 1672 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=NewsRunner:0:1 -s | list 65840 11868 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s | list 62036 10840 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=VirginRunner:0:1 -s | list 58716 5712 /usr/bin/python /var/lib/mailman/bin/qrunner --runner=RetryRunner:0:1 -s This is without hyperkitty or a management interface, just mailman3 core that uses 10 times as much ram as mailman2. Did I do something wrong here? Is there probably a bug in my python environment? Are there any config options that I should set to avoid this huge memory footprint? I found one thread[1] on this mailing list where a user asked a question similar to mine with comparable numbers. best regards, Adi Kriegisch [1] https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…

2 1

Subscription notification sent from the site admins work-mail.
by Henrik Rasmussen 13 Sep '18

13 Sep '18

One of my list administrators told me that all seven list administrators on a list received a "None <foo(a)example.com> has been successfully subscribed to [LIST NAME].", when someone subscribed to their list, however the sender of the mail (From:) was my own work mail address. I am not associated with the list other that I am site admin. The log /opt/mailman/web/logs/smtp.log seems to confirm that Mailman have sendt the mail on behalf of me. Sep 13 09:07:24 2018 (34) <153682964402.36.12950433926619808467@mailman-core> smtp to [LISTNAME](a)lists.example.com for 7 recips, completed in 0.12819194793701172 seconds Sep 13 09:07:24 2018 (34) <153682964402.36.12950433926619808467@mailman-core> post to [LISTNAME](a)lists.example.com from me(a)example.com, 525 bytes How can I, the site admin, be the sender og the mail? Note: Mailman 3 have been running with users half a year, but last week the particular list was imported from Mailman 2 using import21.

2 2

MM3 management tools (for those who need them)
by Dmitry Makovey 13 Sep '18

13 Sep '18

Hi, I wanted to share our current development of MM3-related tools that we currently employ on a daily basis. forks, ideas, pointers and issue reports are welcome: https://gitlab.med.stanford.edu/irt-public/mailman-tools Note that it's a "work in progress" bunch of scripts that work, but YMMV. ...so no refunds ;) -- Sr System and DevOps Engineer SoM IRT

1 0

LookupError: unknown encoding: _iso-2022-jp$esc
by Thor Atle Rustad 07 Sep '18

07 Sep '18

This happens when I open a page of Held Messages containing an offending message. I get Server Error (500) in Postorius. Here is the mailman.log: Sep 07 15:25:39 2018 (32) REST request handler error: Traceback (most recent call last): File "/usr/local/lib/python3.6/wsgiref/handlers.py", line 137, in run self.result = application(self.environ, self.start_response) File "/usr/local/lib/python3.6/site-packages/mailman/database/transaction.py", line 50, in wrapper rtn = function(*args, **kws) File "/usr/local/lib/python3.6/site-packages/mailman/rest/wsgiapp.py", line 214, in __call__ return super().__call__(environ, start_response) File "/usr/local/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__ responder(req, resp, **params) File "/usr/local/lib/python3.6/site-packages/mailman/rest/post_moderation.py", line 160, in on_get resource = self._make_collection(request) File "/usr/local/lib/python3.6/site-packages/mailman/rest/helpers.py", line 159, in _make_collection for resource in collection] File "/usr/local/lib/python3.6/site-packages/mailman/rest/helpers.py", line 159, in <listcomp> for resource in collection] File "/usr/local/lib/python3.6/site-packages/mailman/rest/post_moderation.py", line 150, in _resource_as_dict resource = self._make_resource(request.id) File "/usr/local/lib/python3.6/site-packages/mailman/rest/post_moderation.py", line 77, in _make_resource resource['msg'] = msg.as_string() File "/usr/local/lib/python3.6/site-packages/mailman/email/message.py", line 54, in as_string value = email.message.Message.as_string(self) File "/usr/local/lib/python3.6/email/message.py", line 158, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/local/lib/python3.6/email/generator.py", line 116, in flatten self._write(msg) File "/usr/local/lib/python3.6/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/local/lib/python3.6/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/local/lib/python3.6/email/generator.py", line 243, in _handle_text msg.set_payload(payload, charset) File "/usr/local/lib/python3.6/email/message.py", line 315, in set_payload payload = payload.encode(charset.output_charset) LookupError: unknown encoding: _iso-2022-jp$esc

2 1

confusing contents of database (data corruption?)
by Dmitry Makovey 07 Sep '18

07 Sep '18

I'm troubleshooting slowness and issues in our mailman (container-based) setup. (See also "ORM exceptions" thread) And I've come across really odd contents of DB: mailmandb=# select * from address, (select email, count(email) as instances from address group by email) as emails where emails.instances>1 and emails.email=address.email ; -[ RECORD 1 ]--+---------------------------------------------- id | 1001 email | some_person(a)some.org _original | Some_Person(a)some.org display_name | Some Person verified_on | registered_on | 2018-07-02 03:23:18.761679 user_id | 22543 preferences_id | 146541 email | some_person(a)some.org instances | 2 -[ RECORD 2 ]--+---------------------------------------------- id | 1002 email | some_person(a)some.org _original | display_name | verified_on | registered_on | 2018-07-02 03:23:19.036667 user_id | preferences_id | 146543 email | some_person(a)some.org instances | 2 mailmandb=# select * from member where address_id in (1002,1001); -[ RECORD 1 ]-----+------------------------------------- id | 73682 _member_id | f4cdxxxx-5c64-xxxx-9ac8-1f5xxxxa0f1c role | 4 list_id | mylist.lists.here.stanford.edu moderation_action | address_id | 1002 preferences_id | 146544 user_id | so: 1. address for the same email gets created twice 2. membership references "empty" address 3. membership does not reference proper user 4. preferences_id in all 3 records are different is there a documentation on the DB contents someplace? I feel like our mailman instance slowness is related to DB content but cleaning up that content seems to be risky at best. -- Sr System and DevOps Engineer SoM IRT

1 0

message file gone, but exists in database?
by Thor Atle Rustad 06 Sep '18

06 Sep '18

I'm investigating some problems, and in the process I want to delete some messages. First, listing all messages: >>> messages = list(message_store.messages) Traceback (most recent call last): File "<console>", line 1, in <module> File "/usr/local/lib/python3.6/site-packages/mailman/model/messagestore.py", line 117, in messages yield self._get_message(row) File "/usr/local/lib/python3.6/site-packages/mailman/model/messagestore.py", line 95, in _get_message with open(path, 'rb') as fp: FileNotFoundError: [Errno 2] No such file or directory: '/opt/mailman/var/messages/5X/7W/5X7WMWNORTGVNB6KVWTHHOI2VF2BEQ7K' Since this message should not exist in database, I try to delete it: >>> message_store.delete_message('5X7WMWNORTGVNB6KVWTHHOI2VF2BEQ7K') >>> commit() >>> messages = list(message_store.messages) Traceback (most recent call last): File "<console>", line 1, in <module> File "/usr/local/lib/python3.6/site-packages/mailman/model/messagestore.py", line 117, in messages yield self._get_message(row) File "/usr/local/lib/python3.6/site-packages/mailman/model/messagestore.py", line 95, in _get_message with open(path, 'rb') as fp: FileNotFoundError: [Errno 2] No such file or directory: '/opt/mailman/var/messages/5X/7W/5X7WMWNORTGVNB6KVWTHHOI2VF2BEQ7K' However, it continues to exist in mailman's mind I also cannot delete messages that actually exist. For instance, I find a message in Postorius that I want to delete. I look at the headers and then run >>> message_store.delete_message('6XFISRRFY74NJEE4FTRNWOA2KGKDBZ3T') >>> commit() After that, the message continues to live, both in postorius and as a file: bash-4.3# ls 6XFISRRFY74NJEE4FTRNWOA2KGKDBZ3T What is wrong?

1 0

ORM exceptions
by Dmitry Makovey 06 Sep '18

06 Sep '18

I'm tracing slowness of mailman in our infrastructure, and came across ORM exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2837, in one ret = self.one_or_none() File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2816, in one_or_none "Multiple rows were found for one_or_none()") sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none() During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/wsgiref/handlers.py", line 137, in run self.result = application(self.environ, self.start_response) File "/usr/local/lib/python3.6/site-packages/mailman/database/transaction.py", line 50, in wrapper rtn = function(*args, **kws) File "/usr/local/lib/python3.6/site-packages/mailman/rest/wsgiapp.py", line 214, in __call__ return super().__call__(environ, start_response) File "/usr/local/lib/python3.6/site-packages/falcon/api.py", line 244, in __call__ responder(req, resp, **params) File "/usr/local/lib/python3.6/site-packages/mailman/rest/members.py", line 365, in on_post return self._find(request, response) File "/usr/local/lib/python3.6/site-packages/mailman/rest/members.py", line 389, in _find okay(response, etag(resource._make_collection(request))) File "/usr/local/lib/python3.6/site-packages/mailman/rest/helpers.py", line 159, in _make_collection for resource in collection] File "/usr/local/lib/python3.6/site-packages/mailman/rest/helpers.py", line 159, in <listcomp> for resource in collection] File "/usr/local/lib/python3.6/site-packages/mailman/rest/members.py", line 72, in _resource_as_dict user = member.user File "/usr/local/lib/python3.6/site-packages/mailman/model/member.py", line 128, in user else getUtility(IUserManager).get_user(self._address.email)) File "/usr/local/lib/python3.6/site-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.6/site-packages/mailman/model/usermanager.py", line 88, in get_user return addresses.one().user File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2840, in one "Multiple rows were found for one()") sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one() is this a known thing and what's the best path for fixing it? -- Sr System and DevOps Engineer SoM IRT

1 1

log rotation
by Dmitry Makovey 06 Sep '18

06 Sep '18

I've been looking for a while and cannot find any information regarding log rotation for Mailman3. We're running containerized deployment which complicates the matter slightly, but having "some" approach for log rotation would be helpful. So before I dig into the code looking for SIGUSR1 or SIGHUP handlers can someone point me to a proper documentation for log management with MM3 ? -- Sr System and DevOps Engineer SoM IRT

3 3

ARC
by David Krantz 30 Aug '18

30 Aug '18

Hi, I just found out that gmail uses ARC (sometimes, at least). Is there support for that in Mailman 3? Without having had the time (yet) to read about in depth it it seems as if ARC is intended to solve at least some of the problems that mail lists have with DKIM. Probably causes others... It might be that no support really is needed in mailman so that just milters in sendmail, postfix etc. might do the trick. regards // David

2 1

Migration advice
by Quanah Gibson-Mount 29 Aug '18

29 Aug '18

Hi, I'm looking at migrating from an old server running mailman 2.1.15 (Debian wheezy) to a new Ubuntu 18.04 LTS system. I'd like to upgrade to mailman3 as a part of this migration. I was reading the migration docs (<http://docs.mailman3.org/en/latest/migration.html>), which seem to require both mailman2 and mailman3 be installed concurrently on the same system. Is that correct, or can I rsync the mailman2 list information onto the new server without mailman2 itself being present, and import from there? Thanks! Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

2 1