- Mailman-users - lists.mailman3.org

Database running VERY hot
by Darren Smith March 12, 2018

March 12, 2018

Hello all, We had an old installation of Mailman 2.1 across three servers with over 32,000 mailing lists on them. Due to circumstances beyond my control, we had to shut them down and quickly set up Mailman 3 on a new server. We have Mailman3 set up on a single server this time, as we were having problems with a 3-server setup and Hyperkitty and archives. Long story short, it is up and running, and our users are starting to come back. However, we do not have very many users hitting Postorius and Hyperkitty yet, and we are seeing a distressing number of 504 gateway timeouts. We noticed that there is a significant number of queries to the API whenever hitting Postorius's front page to load the mailing lists, and then on every load. It looks as if it makes a single call to the API to get information about a single list, and does this 10 times - once for each list - to render the page. This gets much worse when users set the view to 200 per page, as they are inclined to do with 32,000 mailing lists... What we've done, actually, is to override the front page and point the users to a different set of static pages where we have all of our lists categorized and searchable, and when they find the list they are looking for, we have a link back to postorius. It has taken a lot of the stress off of the server. However, I still see where the database at times can run VERY hot. We have set up this server to run with PostgreSQL. When I do a ps aux | grep postgres, I find a lot of processes that have been spawned to take care of mailman - a dozen or so. However, one of the threads is taking up about 85% of the load, another thread about 14%, and little dribbles here and there from the rest of the threads. Here are the questions: 1) Has anyone run into problems like these? I would think that maybe this is an issue having so many mailing lists, but honestly there aren't that many people using the system yet. It is going to ramp up as the users realize that we are live again, but we aren't getting that much traffic. 2) Is there any configuration that I can set, especially with regards to the database, so that it can be used more efficiently and not throttle through one or two threads? Please let me know what further information I can provide. Thanks in advance for any help or advice! Silas

5 11

Deactivate list without deleting it?
by Colin Watson March 3, 2018

March 3, 2018

Hi, I'm experimenting with a port of Launchpad's mailing list infrastructure to Mailman 3. If Barry's reading, he may remember that this is no trivial task, so I'm very definitely not promising that it will be done any time soon. :-) However, for the most part it looks relatively tractable with the aid of the plugin improvements in Mailman 3.2.0; I have the rough outline of a plugin that implements much the same feature set as our current pile of monkey-patches against Mailman 2.1 in a much nicer way. One of the major things I haven't figured out yet is list deactivation and reactivation. In our current 2.1-based code, you can deactivate a list in a reversible way, which basically tars up the list directory so that the list archives can be restored later. This isn't a native operation in 2.1, but it could be done because everything lived on the filesystem. In 3, it's less trivial. We could go to some effort to dump things out of the HyperKitty database and restore it on reactivation, but that seems rather complex and a bit silly. Ideally, I'd like mailing lists to have a "deactivated" state in addition to full deletion. In this state, the list would appear nonexistent for most purposes: incoming email would be rejected, Postorius would deny its existence, etc. There'd need to be corresponding extensions to the IArchiver interface which caused the archiver to put its archives of that list into a corresponding deactivated state, in which it would retain the messages but queries for them would return 404 or similar; these extensions would need to be optional, since while it should be relatively easy to do this in HyperKitty it could be hard or impossible for other archivers, and perhaps this should be a separate operation from deactivating the list itself since people may want to close a list while leaving its archives accessible. Has there been any discussion of this in the past, or any advice as to implementation strategy? Thanks, -- Colin Watson [cjwatson(a)ubuntu.com]

3 7

configure Mysql and postfix on mailman3
by ouerheni.nouha＠gmail.com Feb. 20, 2018

Feb. 20, 2018

Hello all , I configured mailman3 following this contributer guide http://docs.mailman3.org/en/latest/devsetup.html but i'am still getting problem with the configuration of MySQL and postfix . Is this section is enough for MySQL configurations ? [database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://myuser:mypassword@mymysqlhost/mailman?charset=utf8&use_unicode=1htmlI will appreciated if there are some documents to provide to cotinue the intergration of mailman3 . Is there any tuto for postfix config ? Thanks in advance Ouerheni Nouha

5 4

import21: A string literal cannot contain NUL (0x00) characters.
by Henrik Rasmussen Feb. 15, 2018

Feb. 15, 2018

Docker-mailman 0.1.1. I am facing errors when using import21 to import list archives. The error message is "A string literal cannot contain NUL (0x00) characters." and the import stops, thus importing only mails up to 2009. The dublicates is mails already imported when retrying. If a mail contains illegal characters I would expect disregarding the particular mail and continue, or ignoring the nul character. bash-4.3# python manage.py hyperkitty_import --verbosity 3 --since "01.01.1970" --list-address example(a)mailman3.ku.dk /opt/mailman-web-data/tmp/example(a)mailman3.ku.dk.mbox Duplicate email with message-id '857A4D84DE6D1D41837DD9284F2729AB08F0C385(a)srv1.example.ku.dk' <857A4D84DE6D1D41837DD9284F2729AB09DA5E45(a)srv1.example.ku.dk> (149) Duplicate email with message-id '857A4D84DE6D1D41837DD9284F2729AB09DA5E45(a)srv1.example.ku.dk' <857A4D84DE6D1D41837DD9284F2729AB09E20126(a)srv1.example.ku.dk> (150) Failed adding message <857A4D84DE6D1D41837DD9284F2729AB09E20126(a)srv1.example.ku.dk>: A string literal cannot contain NUL (0x00) characters. Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line utility.execute() File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 359, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 305, in run_from_argv self.execute(*args, **cmd_options) File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 356, in execute output = self.handle(*args, **options) File "/usr/local/lib/python2.7/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 278, in handle importer.from_mbox(mbfile) File "/usr/local/lib/python2.7/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 152, in from_mbox add_to_list(self.list_address, message) File "/usr/local/lib/python2.7/site-packages/hyperkitty/lib/incoming.py", line 149, in add_to_list email.save() File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 796, in save force_update=force_update, update_fields=update_fields) File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 824, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 908, in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw) File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 947, in _do_insert using=using, raw=raw) File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1043, in _insert return query.get_compiler(using=using).execute_sql(return_id) File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1054, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) ValueError: A string literal cannot contain NUL (0x00) characters. bash-4.3#

6 8

Number of archives in list
by kawhite＠ancestry.com Feb. 15, 2018

Feb. 15, 2018

Hi, I am importing hundreds of thousands of mbox archive files and I want to know if there is a way to call a list and see how many archives got imported / are in the list now (is there a hyperkitty command to gets stats such as number of archives on a list?) Thanks in advance,

2 1

"mailman start" hangs
by Peter Münster Feb. 14, 2018

Feb. 14, 2018

Hi, When starting mailman in a terminal, then it works: --8<---------------cut here---------------start------------->8--- $ mailman start Starting Mailman's master runner $ --8<---------------cut here---------------end--------------->8--- But from "cron", it seems to hang. Output of pstree: --8<---------------cut here---------------start------------->8--- |-cron(3022)---cron(21962)-+-mailman(21986) | `-sendmail(22043)---postdrop(22044) --8<---------------cut here---------------end--------------->8--- So it seems, that mailman does not go into the background, and the email with the output is never sent... How could I solve this issue please? TIA for any hints, -- Peter

3 3

list of subscribers
by Peter Münster Feb. 8, 2018

Feb. 8, 2018

Hi, Where on the web interface is it possible to see the list of subscribers of a mailing list please? TIA for any hint, -- Peter

4 14

Just starting with mailman3
by Loring Feb. 1, 2018

Feb. 1, 2018

I am just starting with mailman 3 where can I find good documentation on getting started? -- Richmond Loring rloring(a)bnl.gov Advanced Systems Engineer

2 1

confirm-email: error NoReverseMatc
by Peter Münster Jan. 31, 2018

Jan. 31, 2018

Hi, When confirming the email address after sign-up, I get this error: --8<---------------cut here---------------start------------->8--- ERROR 2018-01-31 09:31:57,143 28537 django.request Internal Server Error: /accounts/confirm-email/Ng:1egno5:_T5eipR8TTan-zWjLK8EhYuetjQ/ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/views.py", line 283, in post resp = self.login_on_confirm(confirmation) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/views.py", line 332, in login_on_confirm redirect_url=self.get_redirect_url) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/utils.py", line 157, in perform_login get_login_redirect_url(request, redirect_url)) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/utils.py", line 50, in get_login_redirect_url url = url() File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/views.py", line 362, in get_redirect_url self.request) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/adapter.py", line 176, in get_email_confirmation_redirect_url return self.get_login_redirect_url(request) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/adapter.py", line 156, in get_login_redirect_url return resolve_url(url) File "/usr/lib/python2.7/site-packages/django/shortcuts.py", line 205, in resolve_url return urlresolvers.reverse(to, args=args, kwargs=kwargs) File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 579, in reverse return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))) File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 496, in _reverse_with_prefix (lookup_view_s, args, kwargs, len(patterns), patterns)) NoReverseMatch: Reverse for '' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: [] ERROR 2018-01-31 09:31:57,143 28537 django.request Internal Server Error: /accounts/confirm-email/Ng:1egno5:_T5eipR8TTan-zWjLK8EhYuetjQ/ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/views.py", line 283, in post resp = self.login_on_confirm(confirmation) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/views.py", line 332, in login_on_confirm redirect_url=self.get_redirect_url) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/utils.py", line 157, in perform_login get_login_redirect_url(request, redirect_url)) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/utils.py", line 50, in get_login_redirect_url url = url() File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/views.py", line 362, in get_redirect_url self.request) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/adapter.py", line 176, in get_email_confirmation_redirect_url return self.get_login_redirect_url(request) File "/usr/lib/python2.7/site-packages/django_allauth-0.34.0-py2.7.egg/allauth/account/adapter.py", line 156, in get_login_redirect_url return resolve_url(url) File "/usr/lib/python2.7/site-packages/django/shortcuts.py", line 205, in resolve_url return urlresolvers.reverse(to, args=args, kwargs=kwargs) File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 579, in reverse return force_text(iri_to_uri(resolver._reverse_with_prefix(view, prefix, *args, **kwargs))) File "/usr/lib/python2.7/site-packages/django/core/urlresolvers.py", line 496, in _reverse_with_prefix (lookup_view_s, args, kwargs, len(patterns), patterns)) NoReverseMatch: Reverse for '' with arguments '()' and keyword arguments '{}' not found. 0 pattern(s) tried: [] --8<---------------cut here---------------end--------------->8--- What could be the reason, and how could I solve this issue please? Since it's perhaps related, here is my urls.py: --8<---------------cut here---------------start------------->8--- from django.conf.urls import include, url from django.contrib import admin from postorius.views.list import list_index urlpatterns = [ url(r'^postorius/', include('postorius.urls')), url(r'^archives/', include('hyperkitty.urls')), url(r'', include('django_mailman3.urls')), url(r'^accounts/', include('allauth.urls')), url(r'^admin/', include(admin.site.urls)), url(r'^$', list_index, name='list_index'), ] --8<---------------cut here---------------end--------------->8--- TIA for any help, -- Peter

2 4

setup of URLs
by Peter Münster Jan. 29, 2018

Jan. 29, 2018

Hi, I would like to point "https://my-domain.org/" to the list of mailing-lists of this domain. How should I setup my Apache configuration and/or the urlpatterns please? With my actual configuration, the above URL is always rewritten to https://my-domain.org/postorius/lists/ These are my urlpatterns: urlpatterns = [ url(r'^$', RedirectView.as_view( url=reverse_lazy('list_index'), permanent=True)), url(r'^postorius/', include('postorius.urls')), url(r'^archive/', include('hyperkitty.urls')), url(r'', include('django_mailman3.urls')), url(r'^accounts/', include('allauth.urls')), url(r'^admin/', include(admin.site.urls)), ] TIA for any help, -- Peter

4 4