Search results for "Postorius" - Mailman-users

Mailman 3 deployment: ok to use same database instance for core, web, and hyperkitty tables?

by Stephen J. Turnbull

Hi Serban, s.dobrau(a)ucl.ac.uk writes: > We are currently working on a migration plan for Mailman -> Mailman > 3. We have around over 3000 mailing lists with a total of 50K > members Medium-size. I guess a 2-CPU/4GB Linode or similar should handle the Mailman load (including Postorius and HyperKitty), you might want more memory and maybe more CPUs if you're running the RDBMS on the same host. In the 23k lists/50k users/100k inbound messages/day migration described below, we ended up reconfiguring the VMs 3 times before declaring the system stable. IMO the 2 x 8CPU/16GB final setup was way overpowered for the need, but we didn't try to calibrate finer than that. > and 500 worth of archived data. 500 whats? 500MB, 500GB, 500TB? Assuming 500GB, > Our plan is to have core+web (instance 1) and Hyperkitty (instance > 2). With the archives on the FS local to the Hyperkitty. The archives are in the database as BLOBs. There's no reason to have a separate instance for HyperKitty. It's not obvious to me that you need more than that 2x4 Linode for both Mailman and RDBMS, but if you're going to split it's Mailman vs RDMBS, not Mailman vs HyperKitty. > And all 3 tables (core, web, Hyperkitty) pointing to the same > database on a remote SQL (mysql-enterprise) instance. Using class > mailman.database.mysql.MySQLDatabase (and whatever necessary on the > Django/Hyperkitty instance). Make sure your MySQL database(s) for Mailman are configured with the utf8mb4 option for 4 byte UTF-8 support, or it will choke on emojis and the like. I don't think there are any other gotchas for MySQL. Note there are a lot more than 3 tables in Mailman's databases. The "traditional" configuration is a "mailman" database for core and a "mailmanweb" database for Django and the archives, but Mark says that as the tables are disjoint across Mailman, Django, and the archives there's no reason not to use a single "mailman" database for all of them. I believe that's how this list's host is configured. > Is this feasible, Yes. > if so what would be the recommended compute/storage > requirements. I would start with the spec above (2CPU/4GB). If that doesn't perform, bump it to taste. Mailman itself uses about a GB of RAM for all its runner processes. (I'm not a cloud expert, I just assume you can do that the way we did, very simply.) The Python code (including compiled .pycs) is about 300MB. The database storage requirements are nominal for Mailman and Django (maybe 100MB total in the database for a 23k lists, 50k users installation I worked on recently), and I think the database usage for the archives was approximately 1:1 vs. the Mailman2 mboxes. Mark may have a better estimate on that. There is also the full-text search index for the archives. I recommend Xapian, in my experience Whoosh is really slow.[1] I don't recall offhand how big those were but I would assume they're no smaller than 1/10 the size of the mboxes, and maybe quite a bit bigger. Importing list configurations, users, and subscriptions into Mailman 3 is pretty fast, as long as you're not using the stock Postfix support.[2] The problem is that generating the Postfix alias files for the lists seems to be noticably linear in number of lists, which means it's quadratic for the mass import. IIRC with 5000 lists you'd be looking at >1m/list just to keep regenerating Postfix's alias database, ie 5000 minutes. I found two solutions, both of which required patching Mailman. I hope to get both into the next release. I think we got it down to <5s/list, for 5000 lists that's < 1hr. Importing archives is ... yeeech. The client decided they didn't want a ton of archives anyway (you know corporations, if it's not required by law, shred it after 6 months). I don't recall the size estimate accurately but we kept 6 months X 4500 lists, maybe 100GB out of 2.3TB of mboxes. That took 24 hours to import into HyperKitty, without doing the full-text indexing. I do know the original conservative estimate was 20 days to import the whole 2,3 TB. The full-text indexing took more than a week if I remember correctly. The easiest thing to do is to just do no migration, keep the old archive online, and start populating the new archive when you switch over. It's possible to do both the HyperKitty import and full-text indexing incrementally with a bit of planning. It's also quite possible to migrate incrementally, a few lists at a time. Mailman 2 and Mailman 3 can coexist happily on the same host. Mark can advise on that, I think. The only other advice I remember offhand is that Mailman's outgoing smart host should be in the same datacenter as Mailman. We never did debug it, but the system described above with 23k lists had problems with the out queue stalling when smtp.client.com resolved to a different datacenter 500km away. :-) The problem went away as soon as they pinned the MX to the MTAs in the same datacenter. Footnotes: [1] If you're familiar with another full-text search engine, check the django-haystack docs. Haystack supports several other engines, your preference may be among them. [2] Exim4 does not have this problem. I don't know about other supported MTAs.

1 year

Re: Invalid FilterAction: discard. Treating as discard

by Henrik Rasmussen

Mark Sapiro wrote: > I don't understand the question? Are you asking why the log contains > "invalid FilterAction: discard. Treating as discard"? If so, that's a No, that was just log messages I thought would help explain the problem. > If you are asking why the message is empty after filtering and > discarded, It is presumably because it is a text/html message as opposed > to a multipart/alternative message with text/plain and text/html > alternative parts. The Content-Type of the mail sent to the list is multipart/mixed From: SENDER <sender(a)senderdep.example.com> To: "LISTNAME(a)lists.example.com" <LISTNAME(a)lists.example.com> Subject: =?utf-8?B?QW50cm9wb2xvZ2VyIGkgcHJha3NpczogVGlsbWVsZGluZyDDpWJlbg==?= Thread-Topic: =?utf-8?B?QW50cm9wb2xvZ2VyIGkgcHJha3NpczogVGlsbWVsZGluZyDDpWJlbg==?= Thread-Index: AdT/QIb18JA+6hcSTYycFCI+5NuYoQEwPZ/Q Date: Mon, 6 May 2019 13:53:23 +0200 Message-ID: <250E794B36C1E646AF3D81E1F15348C79DA8CA83(a)P2KITMBX06WC03.unicph.domain> Content-Language: en-US X-MS-Has-Attach: yes X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: Content-Type: multipart/mixed; boundary="_006_250E794B36C1E646AF3D81E1F15348C79DA8CA83P2KITMBX06WC03u_" MIME-Version: 1.0 > There are list attributes filter_types and pass_types (also > filter_extensions and pass_extensions, but those probably aren't My Python isn't that great, but this looks like i'm missing something >>> m.filter_content True >>> m.pass_types <generator object pass_types at 0x7f66d407bba0> >>> m.filter_extensions <generator object filter_extensions at 0x7f66d407bdb0> m.pass_extensions <generator object pass_extensions at 0x7f66d407be08> > Presumably, the combination of filter_types and pass_types is not > allowing text/html. I am thinking that as the "Filter content" (postorius) refers to "the settings below", this setting is needed in case list admin wants an explicit reply-to address or change other switches in Alter Messages. But this shouldn't affect the content, even it is text/html, I think, at least not emptying the mail, I think. > You need to examine these settings in mailman shell, e.g. > > list(m.pass_types) >>> list(m.pass_types) ['multipart/mixed', 'multipart/alternative', 'text/plain'] -- Henrik Rasmussen

5 years, 6 months

Re: Two Questions: One General, the Other Specific

by Brian Carpenter

On 2/26/21 12:10 PM, David Wilson wrote: > For context, I have been lurking on this site for a number of years > now, in order to track the progress of this wonderful mailman3 project. > > Further context, I am chair of a 66 unit condominium and pay for two > lists provided by EMWD.com and get a hands on feel from an mailman3 > operations perspective. > > *Is there another site that might best communicate the future goals, > plans, objectives, and time frames for mailman3 or is the status of > the project at the state of being primarily reacting to problems, as > evidenced in this problem solving forum, at least as it seems to me, > with no offense meant or intended?* > > Something I find surprising is a problem described in an image grab > below relating to an inability to amend a list entry to include a name > to be associated with an email address, as if the data needs to be > immutable. When a workaround was unsuccessfully explored to delete > the entry and then resubmit an amended entry but with the same already > deleted email address, that was still refused, presumably because of > pointers to archived data don't permit, but this seems like a > particularly onerous restriction, or perhaps even a bug, or at least a > severe design constraint. *Is this item on an existing to do list?* The server that David's lists is on, is running the latest stable versions of Mailman 3, Postorius, and Hyperkitty in case that helps. @David, we have the issue that you ran into mitigated to a great extent with our Affinity user interface. Just contact me via our support ticket system if you want to explore have your lists migrated to our Mailman 3 Affinity platform. -- Brian Carpenter Harmonylists.com Emwd.com

3 years, 8 months

Re: Changing web access

by Ralf Wiegand

Thank You - that helped. tcp 0 0 192.168.0.27:8001 0.0.0.0:* LISTEN 2054/python tcp 0 0 192.168.0.27:8000 192.168.0.174:50348 ESTABLISHED 2108/python but I am getting : mailmanclient.restbase.connection.MailmanConnectionError: ('Could not connect to Mailman API: ', 'ConnectionError(MaxRetryError("HTTPConnectionPool(host=\'localhost\', port=8001): Max retries exceeded with url: /3.0/lists?advertised=true&count=0&page=1 (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f88fe4cb4e0>: Failed to establish a new connection: [Errno 111] Connection refused\',))",),)') when starting "python manage.py runserver 0.0.0.0:8000" Sorry for all the questions, it isn't easy to put this together. greatly appreciate your help. Ralf On 4/22/20 12:40 PM, Abhilash Raj wrote: > On Wed, Apr 22, 2020, at 7:22 AM, Ralf Wiegand wrote: >> Hello Members, I am testing mailman3 in a test environment. >> >> python manage.py runserver >> Performing system checks... >> >> System check identified no issues (0 silenced). >> April 22, 2020 - 14:13:57 >> Django version 2.2.12, using settings 'settings' >> Starting development server at http://127.0.0.1:8000/ >> Quit the server with CONTROL-C. >> >> tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN >> 6497/python >> tcp 0 0 127.0.0.1:8001 0.0.0.0:* LISTEN >> 6404/python >> >> >> How can I change this to have access from any host on my local network. >> The server running mailman3+hyperkitty+postorius is a rhel7 server >> without a GUI interface. I also need to give access to our developer on >> the same subnet. > Have a look at the docs for `runserver`: > > https://docs.djangoproject.com/en/3.0/ref/django-admin/#runserver > > You basically need to run: > > python manage.py runserver 0.0.0.0:8000 > > >> Thank You, >> >> Best regards, >> >> Ralf Wiegand >> >> _______________________________________________ >> Mailman-users mailing list -- mailman-users(a)mailman3.org >> To unsubscribe send an email to mailman-users-leave(a)mailman3.org >> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ >>

4 years, 7 months

Re: Link to account

by Lars Bjørndal

Thanks a lot. Maybe I've done a mistake, it doesn.t work. At the bottom of /etc/mailman3/settings.py I have: ROOT_URLCONF = 'urls' `cat /etc/mailman3/urls.py` [...] urlpatterns = [ path( '', RedirectView.as_view(url=reverse_lazy('list_index'), permanent=True), ), path('lists/mailman3/', include('postorius.urls')), path('lists/archives/', include('hyperkitty.urls')), path('lists/', include('django_mailman3.urls')), path('lists/accounts/', include('allauth.urls')), path('lists/admin/', admin.site.urls), ] In Apache config for the domain, I now have: <IfModule mod_proxy.c> ProxyPreserveHost On ProxyPass "lists/mailman3" "http://127.0.0.1:8000/lists/mailman3" ProxyPass "lists/archives" "http://127.0.0.1:8000/lists/archives" ProxyPass "lists/accounts" "http://127.0.0.1:8000/lists/accounts" ProxyPass "lists/admin" "http://127.0.0.1:8000/lists/admin" ProxyPass "lists/user-profile" "http://127.0.0.1:8000/lists/user-profile" </IfModule> Restarted mailmanweb, httpd and mailman3. If I go to list.punktskriftutvalget.no/mailman3/, I get a 404 Not Found response in browser. So what's wrong, do you think? Thanks Lars On Tue, Apr 11, 2023 at 10:52:47AM -0700, Mark Sapiro wrote: > On 4/11/23 10:46, Lars Bjørndal wrote: > > Hi, list! > > > > On one of my sites, I want to have all Mailman3 links to go under > > /mailman3. Even though, the Login link goes to a URL like: > > > > https://list.punktskriftutvalget.no/accounts/login/ > > > > How can I ensure it goes to > > https://list.punktskriftutvalget.no/mailman3/accounts/login/ > > > See this thread https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/… > > -- > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, > San Francisco Bay Area, California better use your sense - B. Dylan > > _______________________________________________ > Mailman-users mailing list -- mailman-users(a)mailman3.org > To unsubscribe send an email to mailman-users-leave(a)mailman3.org > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ > Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message… > > This message sent to lars(a)lamasti.net

1 year, 7 months

Re: Hyperkitty performance problem

by Tobias Diekershoff

Hey all, so far we could not find the reason behind the problem. However while reviewing the installation process one one our team mentioned that he read the installation instructions for Hyperkitty different then the rest of the team. Are Mailman3/Postorius and Hyperkitty supposed to run in separated databases, or are they supposed to use the same database? Thanks in advance for clarification! Tobias On 06.11.24 08:18, Tobias Diekershoff wrote: > Hey all, > > last week we caught some AI bots causing high LOAD on our mailman3 > servers archive, that we blocked with nginx rules. > > I'm not sure if it is related to this incident, but we noticed this > when analyzing the log files at the time. In the mailman.log > hyperkitty is listed 10k times and more to archive a single mail over > the time of 2h, seemingly all with the same process ID > > > (1186) HyperKitty archived message [...] > > The mailman3-web.log has matching log entries that the mail was archived > > > hyperkitty.views.mailman Archived message > > every 0.5 seconds. > > In the web interface to the archive, the mail is only listed once though. > > My suspicion is, that these archiving processes are now overloading > the system, preventing a timely catching up work of the queued > messages. I could not find anything in the Mailman3 nor Hyperkitty > documentation to better distribute the system resources to the Mailman > runner or prevent Hyperkitty to restart archiving a mail over and over > again. > > Any hint about where to look for the reason of this is highly > appreciated. > > Best, > Tobias > -- Tobias Diekershoff >>> System Hacker Free Software Foundation Europe Schönhauser Allee 6/7, 10119 Berlin, Germany | t +49-30-27595290 Registered at Amtsgericht Hamburg, VR 17030 | fsfe.org/support OpenPGP-Key ID ... 0x25FE376FF17694A1 Fingerprint ...... 23EE F484 FDF8 291C BA09 A406 25FE 376F F176 94A1

3 days, 23 hours

Re: Changing subscription address

by Stephen J. Turnbull

Abhilash Raj writes: > > There does need to be a way to revert each option to inherit, and > > there currently doesn't seem to be one. The 'Preferences' interface I > > described in the previous post doesn't provide it either. I will > > think about that. > > The implementation of this isn’t very difficult, but the UX of this is > something I don’t know how to implement in that page to be very > honest. Oh, I can play Liz Warren on that one! :-) "I have a plan for that." > We do already have ability to switch from an explicit address to Primary > Address subscription. I would imagine it would be more than just a > button, but more like a dropdown to choose an email or primary address > and switch all subscriptions to that. Allan's situation is that for some reason he has a lot of users (or at least suspects he does) who would be happy to have only one address that there's only one place to change, but for some reason that address is copied to (not inherited by) each of several subscriptions. We should address that, because I'm pretty sure his suspicion is correct. For example, until I changed one of them -- and got two emails about it for my discard pile :-) -- all of my python.org Mailman 3 subscriptions were explicit. I still have about 8 to go. :-) > This is how Core actually does work[1] if you subscribe via Email > and you already have a primary address. It will lookup a user using > the address and if the address is the primary address of the user, > it will subscribe the primary address. If you say so, I guess it's true. Nevertheless, both Allan and I have observed users with only one address, with that address subscribed literally each time to several (~10) lists. > However, if it isn’t the primary address, and this is a new address, it will > subscribe the address instead of user because the address isn’t > verified at this point and hence can’t be a primary address. We could > extend the subscription workflow to add an additional step to set this > address as primary and subscribe the primary address. Yes, please, I think. Users who don't want this for subscriptions to the primary address are going to be (rare) edge cases. If we need an additional attribute on the user to handle the case of an unverified primary address, I think that's an acceptable cost. Note that to "claim" this User as her own, she has to verify that address. And if she *can* verify that address, we can't stop her from doing so, we can just make it more annoying. > Postorius has the full ability to define its own workflow because > REST API provides full expression capability and doesn’t enforce > anything. Previous versions didn’t had support for subscribing as > Primary Address, but that has since changed and is the default one > in the dropdown. > > This is for authenticated subscription, the anonymous subscribe works > similar to email -join command. Mass-subscribe is probably working the > same way. All of these should unify all subscriptions-by-this-address to one User, in my opinion. > Like mentioned above, it should be possible to do what you suggest. I > am getting into the implementation details here, but the Subscription > workflow would need an additional flag that would verify an address > and use the flag to determine if it should subscribe the address or > addresses’s user if an address is provided to the workflow. I don't think so. Just do the lookup, if it succeeds and matches primary address, use inherit. If it doesn't succeed, create a User and use inherit. If it does succeed, and doesn't match, use the explicit address. It seems to me that this flag would be impractical for email subscriptions (email commands with parameters are generally beyond the kind of user this is most important for) or mass subscriptions (how does the list owner know what the user wants?) anyway. If somebody really wants all their subscriptions via primary address to be copies of the primary address rather than user, I'm sorry, but they'll have to deal with that. I can't imagine that it's the usual case, and there's an easy workaround: make your primary address something other than your usual subscription address! If you disagree with me about the relative burden, then maybe we shouldn't do this at all, but I'm pretty sure this is something we should either do for all subscriptions, or for none. > Primary Address is also an attribute of the Mailman User, so it > should be possible for an admin to set that for a user if the user > doesn’t have an account. When they sign up, they’d be able to > manage and update accordingly. True, but AIUI Allan has a *lot* of users and on average they're relatively needy compared to, say, my user population. The point of his request is to smooth workflows for admins with user populations where the admin *wants* to do many things for their users, or to have smooth user workflows when they teach their users to do it for themselves. That's why I'm focusing on the "I have one address and if it changes I want to change it once" case. I think it's both extremely common, and extremely annoying :-) to admins who have to deal with the current situation. Steve

3 years, 8 months

Re: Using REST with CLI using curl .... help needed

by Michael Bakonyi

Dear Stephen, allright, thx a lot for your quick answer! After fiddling around some more time, I guess I had some misunderstandings regarding the REST-API. I guess if we reach the API correctly, we will not be confronted with the CSRF-Error. We will continue to try but nevertheless: What would be a cleaner approach to reach the API from an external IP? I didn't found any examples in the web so far. Cheers, Michael > Am 24.08.2022/ 34 um 14:27 schrieb Stephen J. Turnbull <stephenjturnbull(a)gmail.com>: > > kontakt(a)michaelbakonyi.de writes: > >> I have the same usecase as Krešimir and tried the example of >> Mark. So in mailman.cfg I set hostname to >> "https://lists.mydomain.org" and then fire this command > > I don't recall this, and unfortunately Mark is on vacation and > completely offline until mid-September, except for maybe a couple days > next week. > >> curl -urestadmin:mypw --referer https://lists.mydomain.org --request POST -H 'Content-Type: application/json' --data-binary '{"list_id": "thelist.lists.mydomain.org", "subscriber": "user(a)example.net", "display_name": "Joe User", "pre_verified": true, "pre_confirmed": true, "pre_approved": true}' https://lists.mydomain.org/mailman3/postorius/lists/thelist.lists.mydomain.… >> >> But then I get a CSRF-warning. Do I now have to get the token and >> put it in my POST or is there an easier way to do this? > > As far as *I* know there's no way to avoid the CSRF warning. If that > means you can't get the response data you need, you will have to do > the CSRF two-step. > > With luck maybe Mark will see this in a few days but I think he's > going to be busy preparing the 2d half of his trip. > > Regards, > Steve >

2 years, 2 months

Re: Implicit destination

by David Krantz

On Mon, Aug 27, 2018 at 2:50 AM Mark Sapiro <mark(a)msapiro.net> wrote: > > That is not the issue. Sending the held message notice to a spammer, if > it reaches the spammer, is probably inconsequential. The issue is that > virtually always in spam the sender address is either undeliverable or > is the address of an innocent 3rd party. Sending these notices to > innocent 3rd parties is called backscatter and is a bad thing. > > To avoid this backscatter, you can set the list's > respond_to_post_requests attribute to False. There are two problems > here. One is this setting is not currently exposed in Postorius, and the > other is this will also stop desired held message notices such as those > from list members. Aha. That was a simple solution although with a bad consequence. Will it stop moderation mails to the moderator as well? I'll try. Otherwise it will not be a problem in my case. The lists where I have this problem are more of function addresses for a number of persons (like a board or workgroup) so they accept posts from non-members. I use mailman as a simple forward gets legit mail caught in the receivers spam filters and might cause more black-list problems. You are of course entirely correct in that backscatter is the real problem, I should have thought of that. The thing in the spam I've received lately is somewhat sophisticated: the senders have correct (albeit strange) domains and correct SPF records so they try to seem legit although the senders are located in unexpected parts of the world. I just checked one example, in that case the MX pointed to the sender IP, the SPF was very sloppy (~all) and everything looked correct except for the fact that we usually do not receive mail from Ukraine and the cryptocurrency scam in the body. It almost got caught in the spam filter, but not quite. cheers // David

6 years, 2 months

Re: New releases

by Mark Sapiro

On 6/9/20 1:05 AM, Danil Smirnov wrote: > On Tue, 9 Jun 2020 at 00:05, Mark Sapiro <mark(a)msapiro.net> wrote: > >> After you upgraded did you run these django-admin commands? >> ``` >> django-admin collectstatic --clear --noinput --verbosity 0 >> django-admin compress >> django-admin compilemessages >> django-admin migrate >> ``` >> These 4 commands should be run after every update to Postorius, >> HyperKitty or django_mainlman3. >> > > Hm, interesting... Could you point me to some related docs please? The documentation is weak and should be updated, but see <https://docs.mailman3.org/en/latest/upgrade-3.2.html#migration-steps> > To upgrade, I've just bumped app versions in my Dockerfiles, built and > restarted... > Mailman-core worked right away with no issues. > Mailman-web (Hyperkitty) failed with the error: > > ---------- > ERROR 2020-06-08 19:46:41,942 14 django.request Internal Server Error: > /hyperkitty/ > Traceback (most recent call last): > File "/usr/lib/python3.6/site-packages/compressor/base.py", line 282, in > precompile > mod = import_module(mod_name) > File "/usr/lib/python3.6/importlib/__init__.py", line 126, in > import_module > return _bootstrap._gcd_import(name[level:], package, level) > File "<frozen importlib._bootstrap>", line 994, in _gcd_import > File "<frozen importlib._bootstrap>", line 971, in _find_and_load > File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked > ModuleNotFoundError: No module named 'sassc -t compressed {infile} > {outfile}' ... > I've fixed the error by removing the older file which is > /opt/mailman-web-data/static/hyperkitty/libs/bootstrap/stylesheets/_bootstrap.scss > > After that everything seems to be working fine. I'm not sure about the above error. 'sassc -t compressed {infile} {outfile}' is defined in `settings` COMPRESS_PRECOMPILERS, but it is a command to be run as part of `django-admin compress`, not a module to be imported. Possibly running `django-admin compress` before starting Hyperkitty would have avoided this. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

4 years, 5 months

Mailman-users search results for query "Postorius"