May 2020 - Mailman-users - lists.mailman3.org

Weird hourly indexing error
by Mark Dadgar 11 Jun '20

11 Jun '20

I moved my mail server to a Digital Ocean droplet. I basically installed all the appropriate packages and then rsync’d everything that matters from my metal server up to the cloud instance. Everything more or less works fine. However, I am now getting this hourly error - see below. It is ENTIRELY possible that I was getting this error on the previous server as well, but I didn’t have www-data aliased to root so I would never have seen it. Anyone know what’s going on here? BTW, Digital Ocean is pretty awesome. If anyone wants to try it, ping me for a referral code good for $100 in credit to play with. - Mark ----- mark(a)pdc-racing.net | 408-348-2878 Begin forwarded message: From: root(a)pdc-racing.net (Cron Daemon) Subject: Cron <www-data@mail> [ -f /usr/bin/django-admin ] && flock -n /var/run/mailman3-web/cron.hourly /usr/share/mailman3-web/manage.py runjobs hourly Date: May 5, 2020 at 5:01:00 AM PDT To: www-data(a)pdc-racing.net [ERROR/MainProcess] Failed indexing 1 - 1000 (retry 5/5): Term too long (> 245): XSUBJECT95251413 (pid 36301): Term too long (> 245): XSUBJECT95251413 Traceback (most recent call last): File "/usr/lib/python3/dist-packages/haystack/management/commands/update_index.py", line 97, in do_update backend.update(index, current_qs, commit=commit) File "/usr/local/lib/python3.8/dist-packages/xapian_backend.py", line 488, in update database.replace_document(document_id, document) xapian.InvalidArgumentError: Term too long (> 245): XSUBJECT95251413

3 24

installing mailman3 core
by Ralf Wiegand 29 May '20

29 May '20

Mark - What is the difference between to two install methods? What are my pros and cons? I have been using this: >>>> |# Note: if we are using a dedicated postgresql database server, all postgresql changes need to be done on the actual postgresql server.| |>>>| |sudo yum install postgresql-server postgresql-contrib| |sudo postgresql-setup initdb| |sudo systemctl start postgresql| |sudo systemctl enable postgresql| |sudo -u postgres psql -c ||"SELECT version();"| |sudo useradd -c ||"Mailman User"| |-s /bin/bash -m mailman| |sudo -u postgres create role mailman;| |sudo -u postgres createdb -O $USER mailman| |sudo -u postgres psql -c ||"grant all privileges on database mailman to mailman;"| |sudo -u postgres psql -c ||"ALTER ROLE mailman WITH PASSWORD 'secret123';"| |sudo -u postgres psql -c ||"ALTER ROLE "||mailman||" WITH LOGIN;"| |sudo echo ||"listen_addresses = '*' "| |>> /var/lib/pgsql/10/data/postgresql.conf| |sudo echo ||"# The user mailman will be able to access all databases from a trusted location (any) without a password"| |>> /var/lib/pgsql/data/pg_hba.conf| |sudo echo ||"host all mailman 0.0.0.0/0 trust"| |>> /var/lib/pgsql/data/pg_hba.conf| |sudo systemctl restart postgresql| |sudo ss -nlt | grep 5432| |<<<| |sudo yum install python36* --skip-broken -y| |# check make sure you have both versions| |sudo python --version| |Python 2.7.5| |sudo python3 --version| |Python 3.6.8| |sudo pip3 install psycopg2-binary| |sudo chown mailman.mailman /opt/mailman| |sudo chmod 755 /opt/mailman| |# NOTE: The first initial mailman3 core setup has to be done manually.| |# This is a important step to ensure the user "mailman" will run the actual mailman3 application and not root.| |# switch to user mailman| |mailman# cd /opt/mailman| |mailman# git clone https:||//gitlab.com/mailman/mailman.git| |mailman# cat <<EOT >> /opt/mailman/mailman/src/mailman/config/mailman.cfg| |[devmode]| |enabled: yes| |recipient: root@localhost| |[webservice]| |hostname: localhost| |port: 8001| |show_tracebacks: yes| |use_https: yes| |[mta]| |incoming: mailman.mta.postfix.LMTP| |outgoing: mailman.mta.deliver.deliver| |lmtp_host: localhost| |lmtp_port: 8024| |smtp_host: localhost| |smtp_port: 25| |[database]| |class||: mailman.database.postgresql.PostgreSQLDatabase| |url: postgres:||//mailman:secret123@<IP-ADDRESS or FQDN>/mailman| |EOT| |mailman# python3.6 -m venv venv-3.6| |>>| |mailman# source venv-3.6/bin/activate| |(venv-3.6) [mailman@empire mailman]$ cd mailman| |(venv-3.6) [mailman@empire mailman]$ python setup.py develop| |(venv-3.6) [mailman@empire mailman]$ mailman info| |<<| | | | | |So I was told I don't have to do this, I can just simple do (rhel7) | |yum install pyhton3| |pip install mailman| | | | |

3 4

Hyperkitty issue(s)
by Bryan Loniewski 28 May '20

28 May '20

Hello -- I'm stuck trying to integrate Hyperkitty. I see the following errors in mailman.log and mailmansuite.log respectively: <snipped> [2020-05-27 16:42:11 -0400] [5140] [ERROR] Error handling request /3.1/lists/bry_test4.lists-test.rutgers.edu/archivers Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 134, in handle self.handle_request(listener, req, client, addr) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/gunicorn/workers/sync.py", line 175, in handle_request respiter = self.wsgi(environ, resp.start_response) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/database/transaction.py", line 50, in wrapper rtn = function(*args, **kws) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/rest/wsgiapp.py", line 193, in __call__ return super().__call__(environ, start_response) File "falcon/api.py", line 274, in falcon.api.API.__call__ File "falcon/api.py", line 269, in falcon.api.API.__call__ File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/rest/lists.py", line 406, in on_get for archiver in archiver_set.archivers File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/rest/lists.py", line 407, in <dictcomp> if archiver.system_archiver.is_enabled} File "/opt/mailman/mm/venv/lib/python3.6/site-packages/mailman/model/mailinglist.py", line 587, in system_archiver raise AssertionError('Archiver not found: {}'.format(self.name)) AssertionError: Archiver not found: hyperkitty and <snipped> Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner response = get_response(request) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/postorius/middleware.py", line 42, in __call__ return self.get_response(request) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 36, in inner response = response_for_exception(request, exc) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 90, in response_for_exception response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info()) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 125, in handle_uncaught_exception return debug.technical_500_response(request, *exc_info) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/django/views/debug.py", line 89, in technical_500_response reporter = ExceptionReporter(request, exc_type, exc_value, tb) File "/opt/mailman/mm/venv/lib/python3.6/site-packages/django/views/debug.py", line 254, in __init__ self.template_info = getattr(self.exc_value, 'template_debug', None) File "/opt/mailman/mm/venv/lib/python3.6/tempfile.py", line 479, in __getattr__ file = self.__dict__['file'] KeyError: 'file' Note that I've kinda followed the Virtualenv route along with using Mark's helpful webpage re: his experiences. Some relevant? config files... mailman.cfg: <snip> [archiver.prototype] enable: yes [archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /opt/mailman/mm/mailman-hyperkitty.cfg <snip> mailman-hyperkitty.cfg: [general] base_url: https://a_real_domain/mm3/hyperkitty api_key: <movealong> I can provide additional conf file setups if warranted? I am curious (and perhaps this is in fact the issue) why when I run mailman conf I don't see the archiver.hyperkittty section even though its in the mailman.cfg file? While at the same time I do see archiver.{mail_archive,mhonarc}? I know it's reading my mailman.cfg based on other sections I've already setup/manipulated. Another data point: $ grep -i hyperkitty requirements.txt HyperKitty==1.3.2 mailman-hyperkitty==1.1.0 Any suggestions? ________________________ Bryan Loniewski Rutgers University OIT:EAS:Messaging and Collaboration Services System Administrator

3 7

automation of mailman3 core startup
by Ralf Wiegand 27 May '20

27 May '20

|mailman# source venv-3.6/bin/activate| |(venv-3.6) [mailman@empire mailman]$ cd mailman| |(venv-3.6) [mailman@empire mailman]$ python setup.py develop| |(venv-3.6) [mailman@empire mailman]$ mailman info| | | |Hello Mark -| |Question: The above assumes that the user mailman will run the mailman3 core application. So the user "mailman" has all the correct permissions. The issue I have is to automate this part, so at start up mailman3 will be running, at shutdown, being shutdown properly. | | | |How can I do this?| | | |Ralf |

2 1

Mailman rewrite the Cc header incorrectly
by Alan So 27 May '20

27 May '20

I just notice that if I send an email to a list with a Cc header broken into multiple lines and the pair of double quotes are put in different lines, the email addresses afterwards will be wrongly rewritten. For example, CC: "ABC DEF (XYZ)" <name(a)example.com> It will become the following after going through the mailing list: CC: XYZ <"ABC DEF"@mail.server.name> It seems that Outlook often split the To/Cc header into multiple lines and they often add double quotes to the full name of the users. Is the rewriting of Cc header handled by Mailman's code or another Python class? Thanks. Regards, Alan So

2 21

Emergency Moderation
by Brian Carpenter 26 May '20

26 May '20

Is there an Emergency Moderation function (similar to the one that Mailman 2 had) with Mailman 3 core/Postorius? Thanks, Brian

5 11

Migrating Mailman 3 lists
by Brian Carpenter 26 May '20

26 May '20

Hello Mailman users, I know how to migrate mailman 2 lists to a new mailman 3 server. However I do not know how to migrate a mailman 3 list from one mailman 3 server to another one. Are there any instructions or protocol on how to accomplish this? The Mailman 3 server I want to migrate *from* was setup using Docker. The Mailman 3 server I want to migrate these lists *to* was setup using a python virtual environment via pip. -- Please let me know if you need further assistance. Thank you for your business. We appreciate our clients. Brian Carpenter EMWD.com -- EMWD's Knowledgebase: https://clientarea.emwd.com/index.php/knowledgebase EMWD's Community Forums http://discourse.emwd.com/

9 17

performance report - mailman 3.3.1rc1 / postorius 1.3.1
by sunil soprey 26 May '20

26 May '20

Hi all - I just wanted to share my perf-load results of Mailman3. Hopefully community finds helpful. I realized mailman3 cannot run on AWS t3a* ( 2cpu or 4cpu @ 8 gigs ) Specs: AWS EC2 - m5a.xl and c5.xl Version: mailman 3.3.1rc1 What: Duration load test of 30k emails across 200 lists. Some lists have a depth of 9k users, other lists have 100 emails address. * create 200 lists of random names Results * Postorius kept hanging on vanilla AWS LTS and LTS2 image. I had to tune the kernel settings ( somax, syslimits, tcp wmem) * gunicorn.cfg must be created and workers/threads property must be added. I noticed hanging python and tcp/socket errors * Total run time = 50mins for fresh-load

1 1

Postorius, Hyperkitty new release candidates
by Abhilash Raj 24 May '20

24 May '20

Hi Everyone, I am pleased to announce a couple of more releases: - Postorius: 1.3.3rc1 Changelog: https://docs.mailman3.org/projects/postorius/en/latest/news.html#rc1 Download: https://pypi.org/project/postorius/1.3.3rc1/ - Hyperkitty: 1.3.3rc1 Changelog: https://hyperkitty.readthedocs.io/en/latest/news.html#rc1 Download: https://pypi.org/project/HyperKitty/1.3.3rc1/ - Mailmanclient: 3.3.1rc1 Changelog: https://docs.mailman3.org/projects/mailmanclient/en/latest/src/mailmanclien… Download: https://pypi.org/project/mailmanclient/3.3.1rc1/ - Django-mailman3: 1.3.3rc1 Changelog: https://gitlab.com/mailman/django-mailman3/-/blob/master/README.rst Download: https://pypi.org/project/django-mailman3/1.3.3rc1/ Python 3.6+ and Django 2.0+ is supported for all of them. Django 3.0 support for Hyperkitty requires manually upgrading a dependency (django-haystack>=3.0b2, once a stable version of this has been released, it shouldn't require manually upgrading). There are tons of bug-fixes across the board and some new features. Biggest visible change is switch to Bootstrap 4, which has been long pending for us. Bootstrap 4 completely changes the CSS grid model using the new Flexbox. There might be some small changes or breakages when using on mobile. Please report such issues to us via Gitlab! Some other notable changes are in Postorius, which includes many more list settings exposed include content filtering settings, bounce processing (which was added in the previous release of Mailman Core) settings and some other ones. You can now also specify a reason when rejecting held messages. There was also a gnarly bug, which caused the name of some members to be the string "None". For the longest time, I couldn't figure out the reason for it, but it ended up being a simple fix in mailmanclient's json serialization of display_name, which would result in Python's None value being passed as string "None" to Mailman's API for subscription. There is also better support for filtering visible lists based on the current vhost, which I see a few people are already waiting for from mailman-users list. A full changelog has been added to each project in the top. These are still pre-releases and if you are installing via pip, you would need the `--pre` flag, for e.g. $ pip install --pre --upgrade postorius hyperkitty As with all other releases, these have also been running on mail.python.org and lists.mailman3.org for some time now, so it should be relatively safe to upgrade to. Stable releases for this is now scheduled in 1 week, hopefully, next weekend. -- thanks, Abhilash Raj (maxking)

1 1

Updating from Ubuntu-20.04 LTE installation
by Peter Adebahr 24 May '20

24 May '20

Hi Mark, thank you for the fast solution of our subscription problem - which brings me to the next step: Our installation was done by "apt-get install" in Ubuntu 20, thus still running MM 3.2.2 (and python 3.8.2). Do you have any advise, how to upgrade from there to your current version? I know this might be a bit dangerous, but with today's VMs the risk is much less than it used to be: Do you think it would be possible, just to replace the code and library parts, keeping the database and config files as is? If not, what would you do when you have to keep the system running (except for maintenance time, of course)? Thank you again and and have a nice relaxed weekend Peter

3 2