Search results for "sapiro" - lists.mailman3.org

[MM3-users] Re: Hyperkitty performance problem

by Mark Sapiro

On 12/28/24 05:19, monochromec via Mailman-users wrote: > Following the root cause analysis which Tobias started all those weeks ago we (the admin team behind the installation) are still struggling with the following phenomenon: messages on average take more than 24 hours to be processed, more precisely, the average lifetime of a pickled message object in `/var/lib/mailman3/queue/pipeline` clocks in at around 26 hours). This issue is with Mailman core's pipeline runner. See https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/8-mil… for an overview of message handling and queues in Mailman core. It seems that the pipeline runner(s) are taking an inordinately long time to process a message through the pipeline of handlers causing the pipeline queue to become backlogged. > Couple of stats of the installation: standard installation from Bookworm OS repos, Hyperkitty as archiver and Postorius as web frontend as explained above, running Python 3.11.2 from the standard systemd service as packaged with the Bookworm deb file. All backends (Core + Hyperkitty) are supported by Postgres version 15+248. The MTA is a standard Postfix installation, again from OS packages. The primary support for Debian packages is Debian. See https://wiki.list.org/x/12812344 > The underlying VM has 7 cores with just under 24 GB of main memory. This production instance is handling less than 130 mailing list (MLs) with an average of less than 10 postings per day per ML. CPU core utilisation hovers around 50% with the lion share allocated to the four pipeline runners as part of the MM configuration. This is an average of less than 1 message per minute. Processing a message through the pipeline should take nowhere near a minute. Possibly there is an issue in the rfc_2369.py handler. This handler communicates with HyperKitty via mailman_hyperkitty to obtain values for the List-Archive: and Archived-At: headers added to the message. Possibly there is some configuration issue with mailman_hyperkitty that causes these queries to take a long time and possibly eventually time out. Are there any `Connection to HyperKitty failed:`, `HyperKitty failure on ...` or `Invalid response from HyperKitty` type messages in mailman.log? -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

5 months, 3 weeks

[MM3-users] Re: A little stuck with installation of MM3 - ModuleNotFoundError: No module named 'flufl.lock'

by Mark Sapiro

On 7/26/20 9:12 AM, Odhiambo Washington wrote: > On Sun, 26 Jul 2020 at 18:48, Mark Sapiro <mark(a)msapiro.net> wrote: > >> On 7/26/20 3:15 AM, Odhiambo Washington wrote: >> >>> There is still some confusion on my part about the directives in the file >>> so allow me to seek some clarifications. >>> >>> The following are the contents of a file I have created and placed in my >>> Apache Includes/ directory. >>> I was hoping that with it, I can now access http://lists.my.server/mm3 >> and >>> get the UI. >>> >>> <CUT> >>> >>> # Global section >>> WSGIDaemonProcess mailman-web display-name=mailman-web >>> maximum-requests=1000 umask=0002 user=mailman3 \ >>> group=mailman3 >>> >> python-path=/opt/mailman/mm/venv/lib/python3.7/site-packages:/opt/mailman/mm/venv/lib/python3.7 >>> \ >>> python-home=/opt/mailman/mm/venv home=/opt/mailman/mm/var >>> WSGIRestrictSignal Off >>> >>> <VirtualHost *:80> >>> ServerName lists.my.server >>> ServerAdmin odhiambo(a)gmail.com >>> >>> # (I'm not sure that WSGIRestrictSignal Off is required, but it was in >> the >>> # provided example so I kept it. I also made changes to WSGIDaemonProcess >>> # based on my own mod_wsgi experience elsewhere.) >>> >>> ErrorLog /var/log/myserver-error.log >>> LogLevel debug >>> >>> # This goes in the VirtualHost block for the domain. >>> >>> # Mailman 3 stuff >>> Alias /static "/var/spool/mailman-web/static" *<----- Where is this >>> directory supposed to be and what/who creates it and with what >> permissions?* >> >> >> It is created/updated by the mailman-post-update script based on CONFDIR >> and STATIC_DIR. >> > > The fact that I have run the mailman-post-update script and these > directories aren't created means there is a problem in my setup, yes? If your mailman-post-update script is essentially the same as <https://wiki.list.org/DOC/Mailman%203%20installation%20experience?action=At…> and it ran successfully, it would have created opt/mailman/mm/static and populated it. What was the output when you ran it? > Yes, I have these files: > (venv) [root@gw /opt/mailman/mm]# ls -al /opt/mailman/mm/settings* > -rw-r--r-- 1 mailman3 wheel 15518 Jul 25 14:34 > /opt/mailman/mm/settings.py > -rw-r--r-- 1 mailman3 wheel 15773 Jul 26 10:33 > /opt/mailman/mm/settings_local.py Is there an empty /opt/mailman/mm/__init__.py file? > PS: I made my /opt/mailman to be owned by mailman3:wheel because of a > confusion with the instructions at some point in the reading > of some documentation previously (in my previous attempt last year!) and > had noted that in my NOTES. I'll change this once things get clear for me. > Also, the fact that one of the init scripts > <https://wiki.list.org/DOC/Mailman%203%20installation%20experience?action=At…> > in your "reflection" page was calling sudo :) It does `sudo -u $USER ...` so that it runs Mailman as the Mailman user as it should. Have you set USER=mailman3 in yours? -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

4 years, 10 months

[MM3-users] Re: moving individual list to different domain

by Mark Sapiro

On 3/21/23 06:21, Johannes Rohr wrote: > Dear all, > > on mailman2 it was possible to move a list to a different domain within > the same mailman instance. > > I have found no information on how the same can be accomplished on > mailman3. > > Is this possible at all? In mailman 2.1, a list was identified by its name only without the domain and those names had to be globally unique. A site could be configured to support multiple virtual hosts and a list had a host_name attribute which could be changed if desired to assign the list to a different virtual host, but there couldn't be two lists with the same name even if they had different host names. Mailman 3 is different. In order to remove the restriction that list names (without the domain) had to be globally unique, in MM 3 the domain is part of the list name. I.e., we can now have list(a)example.com and list(a)example.net as two distinct lists in the same installation. Suppose we want to rename list(a)example.com to list(a)example.net. This becomes involved in Mailman 3 because it affects not only Mailman core, but also mail delivery and archiving. Furthermore, the list with fqdn_listname list(a)example.com has the list_id list.example.com. If we change the fqdn_listname to list(a)example.net, the list_id should remain list.example.com per RFC 2919 Sec. 4. <https://www.rfc-editor.org/rfc/rfc2919>. We can just change the list's mail_host attribute via REST <https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/…> or via a database query, but this leads to confusion. I.e., a URL like https://example.com/mailman3/lists/list.example.com/ will still work but https://example.com/mailman3/lists/list@example.com/. To avoid this, we could also change the list_id, but we can't do that via REST as it's a read-only property, and changing the list_id says this is really a new list, not the same list with a new name. Also, none of this addresses archiving. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 years, 2 months

[MM3-users] Re: An archive/read-only list on a full setup?

by Mark Sapiro

On 4/11/23 09:36, Andreas Schamanek wrote: > > I have a working full installation of mailman3, postorius and hyperkitty > (on Debian 11). Now I wanted to host the archive of a remote list, and I > am wondering how to do this properly. > > After I found no documentation and no posting I simply created a list in > mailman3 with only 1 dummy member. The list address is subscribed to the > actual remote list. So, messages arrive and are archived. But that's not > optimal because hyperkitty thinks it's a normal list where it should be > read-only and do not allow subscriptions etc. > > What's the proper way to set up a hyperkitty-only list? There really isn't one as HyperKitty is intended to archive Mailman lists, however, it is possible to have a list in HyperKitty without a corresponding Mailman list. Thus, in your case you could just remove the Mailman 3 list you created and leave the HyperKitty list. Then you have an issue with posts, so you can keep sending the posts to be archived to the prior list address, but arrange for the incoming MTA to pipe mail to that address to a process. That process would do the following. Add a Unix From_ line at the beginning of the message and write the result to a temp file. Invoke a command like django-admin hyperkitty_import --no-sync-mailman --since 1990-01-01 \ -l list(a)example.com /path/to/the/tempfile where list(a)example.com is the actual HyperKitty list. This will archive the message. The addition of a Unix From_ line is to make the message file look like a mbox. Since there is no Mailman list, attempts to post or reply directly from HyperKitty will fail, although such attempts using email software will probably succeed in archiving the mail, and mail directly to that list address will be archived, but such messages won't go to the remote list. This may not be any better than what you're currently doing. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 years, 2 months

[MM3-users] Re: Filter who can send on mailman3 list?

by Marcus Granath

Mark Sapiro wrote: > and to create a non-member for no-reply(a)abc.com and set it's > moderation action also to Defer or Default Processing." Thanks Mark Sapiro, now I got the "Moderation Action" set correct for the real users. BUT, I'm sorry to say that I'm still confused regarding how to add the "non-member" user ("no-reply(a)abc.com") - can you please guide some more here also? ;-) The whole idea (the Use Case) here is that we want our list to be an "announcement" list that will send announcement and the subscribers should not be able to send announcements themself - and usually the "announcements" are sent out with an e-mail address (i.e "no-reply(a)abc.com") that I just "dress" with a "special" name and "no-reply" address.. ;-)

4 years, 2 months

[MM3-users] Re: /hyperkitty/list/ shows "example.com" / mailman3 archives not updated

by Mark Sapiro

On 5/17/20 6:05 AM, patrick-mailinglists(a)whonix.org wrote: > /hyperkitty/list/ shows "example.com". Not the actual domain name that I configured. How can I fix this? Ensure that your actual domain is listed in the Django admin interface under Sites. Then ensure that your SITE_ID setting in your Django settings is correct. This is tricky because the site id might not be what you expect. You have to look in your database at the django_site table[1]. This table contains one row per site with columns id, domain and name, and the id values don't necessarily match what you'd think from looking at Sites in the Django admin interface. > I would not mind too much about hyperkitty showing "example.com". But the more important issue is that mailman3 archives are not updated. Therefore I guess it's a symptom of a more important issue. I doubt that these things are related. > E-mails to the list are delivered to e-mail subscribers, though. Seems only the web part is broken. What's in Mailman's log (var/logs/mailman.log) and are the messages all queued in var/archivers/hyperkitty/spool? If so, mailman-hyperkitty can't communicate with hyperkitty. Ensure that the settings in mailman-hyperkitty.cfg are correct. base_url: needs to be a URL that can access hyperkitty and api_key needs to match the MAILMAN_ARCHIVER_KEY setting in your Django settings except that it's quoted in the Django settings and not in mailman-hyperkitty.cfg. > Using Debian buster mailman3 packages. In configuration folder /etc/mailman3 I cannot find any textual string saying "example.com". > (`grep -r -i example /etc/mailman3`) > Therefore I don't know where mailman3 gets "example.com" from. Please see <https://wiki.list.org/x/12812344> [1] other ways to get this are a) increment the SITE_ID setting until you find the one that works or b) in a django admin shell do from django.contrib.sites.models import Site for site in Site.objects.all(): print('id: {}, name: {}, domain: {}'.format( site.id, site.name, site.domain)) -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

5 years, 1 month

[MM3-users] Re: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

by Mark Sapiro

On 10/13/20 1:11 AM, r.woithe(a)callassoftware.com wrote: > Hello, > System: > Mailman Core Version GNU Mailman 3.2.2 (La Villa Strangiato) > Mailman Core API Version 3.0 > Mailman Core Python Version 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0] > > all this runs on Ubuntu 20.04. > Our mailman3-web.log file is full of this error.Traceback: > > 00:00:17 [Q] ERROR reincarnated pusher Process-1:731365 after sudden death > 00:00:17 [Q] INFO Process-1:731366 pushing tasks at 3061233 > Process Process-1:731366: > Traceback (most recent call last): > File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap > self.run() > File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run > self._target(*self._args, **self._kwargs) > File "/usr/lib/python3/dist-packages/django_q/cluster.py", line 300, in pusher > task = SignedPackage.loads(task[1]) > File "/usr/lib/python3/dist-packages/django_q/signing.py", line 28, in loads > return signing.loads(obj, > File "/usr/lib/python3/dist-packages/django_q/core_signing.py", line 38, in loads > return serializer().loads(data) > File "/usr/lib/python3/dist-packages/django_q/signing.py", line 44, in loads > return pickle.loads(data) > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128) > > Is there a patch or workaround for this error or we are the one and only? This is a qcluster issue. Without knowing what it is that qcluster is looking at that is causing this, I have no idea what the problem is. You could try disabling qcluster. You probably have something like ``` Q_CLUSTER = { 'timeout': 300, 'save_limit': 100, 'orm': 'default', } ``` in settings.py. You can comment out or remove this or I think you can just put `del Q_CLUSTER` in settings_local.py. Possibly this will avoid the error or move it somewhere else where we can get more information. Possibly you can also get more information on the error by implementing a simple error reporter for qcluster. See <https://django-q.readthedocs.io/en/latest/errors.html>. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

4 years, 7 months

[MM3-users] Re: Installing mailman 3

by Rafael Mora

YEEEES! At last!!! I can see the web UI. Now I´m getting this message in the web UI: Something went wrong Mailman REST API not available. Please start Mailman core. Where do I configure the REST API ????? Thank you El vie., 21 jul. 2017 a las 16:32, Mark Sapiro (<mark(a)msapiro.net>) escribió: > On 07/21/2017 02:27 PM, Rafael Mora wrote: > > > > it says it needs whoosh > > > > How do I install that? on env2 with python2 or env with python3 ??? > > > In env2 do 'pip install whoosh' > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list > mailman-users(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > -- Atentamente / Best Regards Ing. Rafael Mora

7 years, 10 months

[MM3-users] Re: Mails are not archived at all -- how to check if Hyperkitty is running or not?

by Franklin Weng

Yes, it is checked. Mark Sapiro <mark(a)msapiro.net> 於 2019年10月30日週三 11:19 寫道： > I haven't read this thread in detail. I've been off line for 3 days due > to a Public Safety Power Shutoff, but my first question is if you go the > the Pustorius web UI for the list(s) and look at Settings -> Archiving > -> Active archivers, is hyperkitty checked? > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ >

5 years, 7 months

[MM3-users] Re: Shorted Subjects in Mailman

by Mark Sapiro

On 7/26/22 03:44, Stephan Krinetzki wrote: > Victoriano Giralt wrote: >> On mar, 2022-07-26 at 10:09 +0000, Stephan Krinetzki wrote: >>> Hi, >>> Hi, >>> i tried now with a Postfix 3.x -> Mailman >>> Postfix has en extremely useful setting called debug_peer_list where >> you can list addresses of machines whose messages you are interested in >> and the level of detail can be set with debug_peer_level. this will >> result on very detailed logging of the SMTP conversation. >> But, before resorting to this very invasive method, I'd put messages on >> the HOLD queue and inspect them with postcat, before releasing them to >> be passed to Mailman. this way, you will be able to see if the subjects >> are cur "before" reaching Mailman. >>> Original to Mailman: https://pastebin.com/c4BHeEkJ >>> Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- >> LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= >>> After Recieving: https://pastebin.com/JG0NKbjq >>> Subject: =?utf-8?q?=5BJaninsliste=5D_=5BSAV- >> LINUX=5D_Notice_from_Sophos_Anti-Virus_on_server=2Eexample=2Ecom?= >> Both of then show the same Subject: header. > > In https://pastebin.com/c4BHeEkJ, Line 79+80 it says > > `Subject: [SAV-LINUX] Notice from Sophos Anti-Virus on server.example.com > [DEMO1]` In the original message you posted at https://pastebin.com/c4BHeEkJ, the continuation line `[DEMO1]` is not indented and is thus not a continuation of the Subject: header but is a separate line. It appears that several other continuation lines are also not indented, so I can't tell what the original message to Mailman actually looked like. However, I did a test injecting directly to Mailman a message with ``` Subject: [SAV-LINUX] Notice from Sophos_Anti-Virus on server.example.com [DEMO1] ``` on a German language test list with subject prefix [Janinsliste] and I get the same result, I.e. the RFC 2047 encoded ``` Subject: =?utf-8?q?=5BJaninsliste=5D=5BSAV-LINUX=5D_Notice_from_Sophos=5FAnti-Virus_on_server=2Eexample=2Ecom?= ``` so this is definitely a Mailman bug (or a bug in the underlying Python email package. Thank you for the report. I have created https://gitlab.com/mailman/mailman/-/issues/1021 for this issue. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 years, 10 months

Search results for query "sapiro"