Re: Changing subscription address
by Abhilash Raj
> On Mar 22, 2021, at 7:53 AM, Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp> wrote:
>
> Abhilash Raj writes:
>
>> Well a couple of stars have to align for it to work unfortunately.
>
> I'm not going to wait for the stars; I have backward-incompatible
> surgery in mind.
>
>> Add to that you can’t set an address as primary using just email
>> commands, you end up with what you are seeing.
>
> Right. The idea is that the first time an address is used, a User
> will be created and that address will be primary for that User. When
> the person wants to claim that address, they go to Postorius, create a
> Postorius account, do the address verification dance and set
> credentials, which links the Postorius account to the User via the
> Address.
>
> This is why the ability to merge Users becomes an urgent need: people
> who subscribe via email are going to end up with multiple Users.
Well, it doesn’t change the situation much from status quo. You also
have multiple users today, even if you subscribe via individual address
since an address always has a User. And like we were discussing over
on the other thread, when the owner decides to join in P and control both the
addresses from single P account, we’ll merge both users.
>
>> We don’t necessarily need the flag on user to set an unverified primary
>> address, we might be able to get away without that. We could set the
>> address as primary *after* we verify it. We always verify addresses
>> regardless of the choice of the subscription mode (user or address).
>
> I'm not clear what you're saying. In general admins can't read your
> mail, so they can't verify you on a mass subscribe or an import.
What I meant was, we can set the address as primary after the verification
dance with the user and then subscribe the primary address.
The existing subscription workflow is a multi-step process that can be
paused and resumed. We pause when we send out a confirmation
email and then resume when that confirmation token is somehow
processed via email or API. So, post receiving the confirmation and
before actually subscribing, we can se the address as primary if the
user doesn’t have one and subscribe user as inhert. We don’t do any
of that if user already has chosen a primary address and it doesn’t
match the current address.
Also, mass-subscribe and import does allow you to verify an address by
clicking this “Pre-verified” checkbox, which can be useful if you are
migrating from a different service or you exported a list of users from
somewhere you know to be valid.
>
>> Right now, P is explicit about it. You can choose if you want Primary
>> Address or one of the explicit addresses added to your account. The
>> primary address is the pre-selected choice, so if you go and just click
>> on subscribe button, you get the primary address being subscribed. If
>> you choose the explicit address then we subscribe the explicit address.
>>
>> Is that something you think needs modifying?
>
> No. The option to set a subscription to the literal primary address
> should still be available. I don't see a good reason to complicate
> the UI logic by comparing primary to literal addresses just to remove
> that one. I just want the primary address to be populated
> automatically, and inherit-from-primary be the default subscription
> address when the subscription address matches primary, *unless* the
> subscriber explicitly chooses that literal address in Postorius.
>
> This is backward-incompatible, but I think it will be overwhelmingly
> popular with new users and old.
There would be no behavioral change for users using email commands.
For those using P, I think will be able to manage and switch rather easily.
>
>> The email join command does the first part you suggested, (if the lookup
>> succeed and matches primary address, use inherit) but doesn’t do the
>> second part (it will create a user, add the address to it and then use the
>> address to subscribe). We can and should fix that.
>
> I think that's all we need to do going forward.
>
> We do need a better UI for setting options (including address) for a
> selection of subscriptions, but that's an independent task.
>
> Thanks for looking at this, Abhilash!
Of course :)
>
> Steve
--
thanks,
Abhilash Raj (maxking)
4 years, 3 months
Re: Intermittent: pymysql.err.InternalError: Packet sequence number wrong - got 102 expected 77
by Mark Sapiro
For anyone who's interested, here's a reformatted version of the traceback.
```
> Full log follows:
> [2021-05-23 07:31:45 +0200] [8244] [ERROR] Error handling request /3.1/lists?advertised=true&count=10&page=1
> Traceback (most recent call last):
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
> cursor, statement, parameters, context
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
> cursor.execute(statement, parameters)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
> result = self._query(query)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
> conn.query(q)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query
> self._affected_rows = self._read_query_result(unbuffered=unbuffered)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
> result.read()
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1082, in read
> self._read_result_packet(first_packet)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1151, in _read_result_packet
> self._get_descriptions()
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1222, in _get_descriptions
> field = self.connection._read_packet(FieldDescriptorPacket)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 671, in _read_packet
> % (packet_number, self._next_seq_id))
> pymysql.err.InternalError: Packet sequence number wrong - got 102 expected 77
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/database/transaction.py", line 50, in wrapper
> rtn = function(*args, **kws)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/wsgiapp.py", line 193, in __call__
> return super().__call__(environ, start_response)
> File "falcon/api.py", line 274, in falcon.api.API.__call__
> File "falcon/api.py", line 269, in falcon.api.API.__call__
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/lists.py", line 322, in on_get
> resource = self._make_collection(request)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/helpers.py", line 172, in _make_collection
> request, self._get_collection(request))
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/helpers.py", line 167, in _paginate
> return list_start, total_size, collection[list_start:list_end]
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/utilities/queries.py", line 42, in __getitem__
> return self._query[index]
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__
> return list(res)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
> return self._execute_and_instances(context)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances
> result = conn.execute(querycontext.statement, self._params)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute
> return meth(self, multiparams, params)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
> return connection._execute_clauseelement(self, multiparams, params)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement
> distilled_params,
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context
> e, statement, parameters, cursor, context
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception
> sqlalchemy_exception, with_traceback=exc_info[2], from_=e
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
> raise exception
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context
> cursor, statement, parameters, context
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
> cursor.execute(statement, parameters)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute
> result = self._query(query)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query
> conn.query(q)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query
> self._affected_rows = self._read_query_result(unbuffered=unbuffered)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
> result.read()
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1082, in read
> self._read_result_packet(first_packet)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1151, in _read_result_packet
> self._get_descriptions()
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1222, in _get_descriptions
> field = self.connection._read_packet(FieldDescriptorPacket)
> File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 671, in _read_packet
> % (packet_number, self._next_seq_id))
> sqlalchemy.exc.InternalError: (pymysql.err.InternalError) Packet sequence number wrong - got 102 expected 77 [SQL: SELECT mailinglist.list_id AS mailinglist_list_id, mailinglist.preferred_language AS mailinglist_preferred_language, mailingli$ FROM mailinglist WHERE mailinglist.advertised = true ORDER BY mailinglist.list_id LIMIT %(param_1)s] [parameters: {'param_1': 10}]
> (Background on this error at: http://sqlalche.me/e/2j85)
```
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
4 years, 1 month
Re: How To Install Mailman 3 on Debian 10 (Complete Guide)
by Allan Hansen
My 2 cents’ worth:
When moving to MM3 from MM2, the latter of which I (relatively) easily installed myself, I read what I could find and went on installing MM3. I soon ran into trouble,
namely somewhere around the hooking up of the web server. I ran into a thicket that I could not decipher.
Then I asked the person who now hosts the server. He soon ran into trouble at around the same place (as far as I remember).
I then asked someone who had been very helpful with MM2 to do the installation. He's a professional. He soon ran into trouble. I don’t know where, but he tried over several weeks to get it going.
The trouble?
Finding the correct documentation.
Understanding the documentation, which is not exactly clear.
Doing what the documentation says, if it says it, and things not working as they should.
I then asked Brian, who has helped a lot of people (like many of you have). He did the installation very quickly and with no trouble.
Don’t tell him this, but it was money well spent! :-)
Yours,
Allan Hansen
hansen(a)rc.org
> On Feb 25, 2021, at 18:38 , Brian Carpenter <brian_carpenter(a)emwd.com> wrote:
>
> On 2/25/21 8:28 PM, Abhilash Raj wrote:
>> Thanks Brian, this looks quite comprehensive in the details. The only
>> thing I am a bit concerned about is granting sudo privileges to the
>> Mailman user. It really shouldn't have sudo given that Mailman and
>> Django are supposed to run as mailman user. Any compromise of
>> the Django application will provide the attacker root on the machine.
>
> Thanks for pointing that out. I have removed that step. It really wasn't needed.
>
>> Is there a specific reason that you chose to go with an entirely new doc
>
> Yes. Just from observing the communications on this list, there are multiple ways to install Mailman 3 which produces installation issues, database problems, outdated apps, etc. My document produces a VERY stable installation of Mailman 3 and one that is easy to maintain and update. It is also very comprehensive covering not just Mailman Suite, but the configuration of an entire server environment. Personally I think the choice of Debian, NGINX, Postfix, and PostgreSQL makes for a long and loving relationship with Mailman 3 but I do understand everyone has their own environment that they want to work with. My guide represents a very stable and easily maintained environment for me and hopefully others.
>
>> rather than helping to improve the existing one? Several parts of it (at
>> least the ones that official guide covers) seems similar to me and is
>> duplicate information that at least two people are going to spend time
>> writing and maintaining in future.
>
> Is there an official guide? See that is the problem. I did not know there was an official guide. Because every time I tried finding documentation in the past using Google searches, I got fed outdated information. So for me the guide that I came up with, works great for me and if no one else uses it, I certainly will.
>
> Your guide, by nature has to be minimalistic to accommodate all the various server environments out there. My is meant to be narrow but comprehensive.
>
>
>> Is it something about the contribution process to the official documentation
>> that makes it hard for people to contribute? Most of the pages at
>> docs.mailman3.org or come from this[1] repo and use Sphinx to build and
>> REsT formatting (.rst).
>>
>> [1]:https://gitlab.com/mailman/mailman-suite-doc
>
> Well I don't know much about any of the above approach to documentation. So I would say that is a hindrance.
>
>
>> I am just trying to understand how can we lower the barrier for community
>> members to help contribute to existing docs instead of them having to create
>> new ones. Specifically around installation, since that tends to get stale often
>> when depedent packages change or a new dependency is added that
>> breaks the installation.
>
> I had to update my guide twice due to recent changes made to the installation process such as the addition of the Dart Sass section and the problem with the cryptography module when installing Hyperkitty. While my guide covers these new hurdles to a new installation, I don't think yours says anything about them. The Dart Sass section came from Mark's comments on this list.
>
> --
> Brian Carpenter
> Harmonylists.com
> Emwd.com
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
4 years, 4 months
Re: Postorius no connection to REST API
by Richard Rosner
Mark Sapiro wrote:
> > You are in a better position to answer that than am I.
> What does sudo netstat -lntp show?
A lot. But since most of that isn't relevant here:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN 20241/master
tcp 0 0 127.0.0.1:8024 0.0.0.0:* LISTEN 14076/python3
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 20241/master
tcp 0 0 127.0.0.1:8001 0.0.0.0:* LISTEN 14080/python3
tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 20241/master
tcp6 0 0 :::80 :::* LISTEN 13882/apache2
tcp6 0 0 :::465 :::* LISTEN 20241/master
tcp6 0 0 :::25 :::* LISTEN 20241/master
tcp6 0 0 :::443 :::* LISTEN 13882/apache2
tcp6 0 0 :::587 :::* LISTEN 20241/master
> What does ps -fwwa|grep rest show?
root 15055 14843 0 12:58 pts/1 00:00:00 grep rest
So whatever it's supposed to find, it's not there
> > mailman3-web.service must also run as list.
I changed that. It didn't like it.
systemctl status mailman3-web.service
● mailman3-web.service - Mailman3-web uWSGI service
Loaded: loaded (/lib/systemd/system/mailman3-web.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-08-11 13:05:39 CEST; 32s ago
Docs: file:///usr/share/doc/mailman3-web/README.rst
Process: 15570 ExecStart=/usr/bin/uwsgi --plugin python3 --ini /etc/mailman3/uwsgi.ini (code=exited, status=1/FAILURE)
Main PID: 15570 (code=exited, status=1/FAILURE)
Status: "initializing uWSGI"
Aug 11 13:05:39 mail systemd[1]: Starting Mailman3-web uWSGI service...
Aug 11 13:05:39 mail systemd[1]: mailman3-web.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 13:05:39 mail systemd[1]: mailman3-web.service: Failed with result 'exit-code'.
Aug 11 13:05:39 mail systemd[1]: Failed to start Mailman3-web uWSGI service.
Aug 11 13:05:39 mail systemd[1]: mailman3-web.service: Service RestartSec=100ms expired, scheduling restart.
Aug 11 13:05:39 mail systemd[1]: mailman3-web.service: Scheduled restart job, restart counter is at 5.
Aug 11 13:05:39 mail systemd[1]: Stopped Mailman3-web uWSGI service.
Aug 11 13:05:39 mail systemd[1]: mailman3-web.service: Start request repeated too quickly.
Aug 11 13:05:39 mail systemd[1]: mailman3-web.service: Failed with result 'exit-code'.
Aug 11 13:05:39 mail systemd[1]: Failed to start Mailman3-web uWSGI service.
> > So, what do you have in your apache config for proxying to uwsgi and
> what's your uwsgi configuration.
lists-ssl.conf:
<VirtualHost *:443>
ServerAdmin admin(a)domain.de
ServerName lists.domain.de
Alias /mailman3/favicon.ico /var/lib/mailman3/web/static/postorius/img/favicon.ico
Alias /mailman3/static /var/lib/mailman3/web/static
<Directory "/var/lib/mailman3/web/static">
Require all granted
</Directory>
<IfModule mod_proxy_uwsgi.c>
ProxyPass /mailman3/favicon.ico !
ProxyPass /mailman3/static !
ProxyPass "/mailman3" "unix:/run/mailman3-web/uwsgi.sock|uwsgi://localhost:8001/"
</IfModule>
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
SSLCertificateFile /etc/ssl/certs/lists.domain.de.cert.pem
SSLCertificateKeyFile /etc/ssl/private/lists.domain.de.private.pem
SSLCertificateChainFile /etc/ssl/certs/dfnca.pem
SSLCACertificateFile /etc/ssl/certs/rwthcert.pem
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
Protocols h2 http/1.1
SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256
SSLHonorCipherOrder off
#RewriteEngine on
#RewriteRule ^/$ https://lists.domain.de/listinfo
Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
Header always set X-Frame-Options: "SAMEORIGIN"
Header always set X-Xss-Protection "1; mode=block"
Header always set X-Content-Type-Options "nosniff"
Header always set Content-Security-Policy "default-src 'self' *.domain.de; script-src 'self' *.domain.de; connect-src 'self' *.domain.de; img-src 'self' *.domain.de; style-src 'self' 'unsafe-inline' *.domain.de; object-src 'self' *.domain.de; frame-src 'self' *.domain.de;"
Header always set Referrer-Policy "no-referrer-when-downgrade"
</VirtualHost>
<VirtualHost *:80>
RewriteEngine On
RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>
I guess with uwsgi config you mean the /etc/mailman3/uwsgi.ini file?
[uwsgi]
# Port on which uwsgi will be listening.
uwsgi-socket = /run/mailman3-web/uwsgi.sock
#Enable threading for python
enable-threads = true
# Move to the directory wher the django files are.
chdir = /usr/share/mailman3-web
# Use the wsgi file provided with the django project.
wsgi-file = wsgi.py
# Setup default number of processes and threads per process.
master = true
process = 2
threads = 2
# Drop privielges and don't run as root.
uid = www-data
gid = www-data
plugins = python3
# Setup the django_q related worker processes.
attach-daemon = python3 manage.py qcluster
# Setup hyperkitty's cron jobs.
#unique-cron = -1 -1 -1 -1 -1 ./manage.py runjobs minutely
#unique-cron = -15 -1 -1 -1 -1 ./manage.py runjobs quarter_hourly
#unique-cron = 0 -1 -1 -1 -1 ./manage.py runjobs hourly
#unique-cron = 0 0 -1 -1 -1 ./manage.py runjobs daily
#unique-cron = 0 0 1 -1 -1 ./manage.py runjobs monthly
#unique-cron = 0 0 -1 -1 0 ./manage.py runjobs weekly
#unique-cron = 0 0 1 1 -1 ./manage.py runjobs yearly
# Setup the request log.
#req-logger = file:/var/log/mailman3/web/mailman-web.log
# Log cron seperately.
#logger = cron file:/var/log/mailman3/web/mailman-web-cron.log
#log-route = cron uwsgi-cron
# Log qcluster commands seperately.
#logger = qcluster file:/var/log/mailman3/web/mailman-web-qcluster.log
#log-route = qcluster uwsgi-daemons
# Last log and it logs the rest of the stuff.
#logger = file:/var/log/mailman3/web/mailman-web-error.log
logto = /var/log/mailman3/web/mailman-web.log
3 years, 10 months
Intermittent: pymysql.err.InternalError: Packet sequence number wrong - got 102 expected 77
by Torge Riedel
|Hi list, sometimes I get the following error in mailman core's log: "pymysql.err.InternalError: Packet sequence number wrong - got 102 expected 77". I attached the full log after this mail. I check the provided link in the log (|||http://sqlalche.me/e/2j85|) and also searched for it on the internet and found the following possible reasons: 1. DB user does not have permissions at all -> I doubt that, because in general mailman is working ok 2. DB user is not allowed to connect from remote -> I doubt that too, because everything is on one host and DB user is allowed to connect from localhost 3. Multi-Threading -> pymysql is mentioned to be not thread-safe and each thread needs its own connection May this be caused by multi-threading? What do I have to do (is there anything I can do?) to solve this on my own?|||||
Kind regards
Torge
||
|Full log follows: [2021-05-23 07:31:45 +0200] [8244] [ERROR] Error handling request /3.1/lists?advertised=true&count=10&page=1 Traceback (most recent call last): File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context cursor, statement, parameters, context File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute cursor.execute(statement, parameters) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute result = self._query(query) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query conn.query(q) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read() File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1082, in read self._read_result_packet(first_packet) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1151, in _read_result_packet self._get_descriptions() File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1222, in _get_descriptions field = self.connection._read_packet(FieldDescriptorPacket) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 671, in _read_packet % (packet_number, self._next_seq_id)) pymysql.err.InternalError: Packet sequence number wrong - got 102 expected 77 The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/database/transaction.py", line 50, in wrapper rtn = function(*args, **kws) File
"/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/wsgiapp.py", line 193, in __call__ return super().__call__(environ, start_response) File "falcon/api.py", line 274, in falcon.api.API.__call__ File "falcon/api.py", line 269, in falcon.api.API.__call__ File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/lists.py", line 322, in on_get resource = self._make_collection(request) File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/helpers.py", line 172, in _make_collection request, self._get_collection(request)) File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/rest/helpers.py", line 167, in _paginate return list_start, total_size, collection[list_start:list_end] File "/opt/mailman/core/venv/lib/python3.6/site-packages/mailman/utilities/queries.py", line 42, in __getitem__ return self._query[index] File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__ return
list(res) File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__ return self._execute_and_instances(context) File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3506, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1020, in execute return meth(self, multiparams, params) File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_clauseelement distilled_params, File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1324, in _execute_context e, statement, parameters, cursor, context File
"/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1518, in _handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from_=e File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_ raise exception File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1284, in _execute_context cursor, statement, parameters, context File "/opt/mailman/core/venv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute cursor.execute(statement, parameters) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 170, in execute result = self._query(query) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/cursors.py", line 328, in _query conn.query(q) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 517, in query self._affected_rows =
self._read_query_result(unbuffered=unbuffered) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 732, in _read_query_result result.read() File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1082, in read self._read_result_packet(first_packet) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1151, in _read_result_packet self._get_descriptions() File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 1222, in _get_descriptions field = self.connection._read_packet(FieldDescriptorPacket) File "/opt/mailman/core/venv/lib/python3.6/site-packages/pymysql/connections.py", line 671, in _read_packet % (packet_number, self._next_seq_id)) sqlalchemy.exc.InternalError: (pymysql.err.InternalError) Packet sequence number wrong - got 102 expected 77 [SQL: SELECT mailinglist.list_id AS mailinglist_list_id, mailinglist.preferred_language AS
mailinglist_preferred_language, mailingli$ FROM mailinglist WHERE mailinglist.advertised = true ORDER BY mailinglist.list_id LIMIT %(param_1)s] [parameters: {'param_1': 10}] (Background on this error at: http://sqlalche.me/e/2j85) |
4 years, 1 month
Re: Easiest way to install a new mailman3 deployment?
by Andrew Hodgson
Hi,
I completed this on a Debian Buster server instance during last year and have got some notes on the process. I decided to go down the venv route rather than use Docker, but I already had it up and running using the Docker instance on another machine. I found getting it up and running on Docker helped me understand how everything hung together, but ultimately I wanted to run it under a venv so I could understand it a bit more and also I wasn't using Docker in the rest of the environment on that server. I also got caught out earlier on with the Docker containers changing the search engine from Woosh to Xapian which is why I wanted more control over what was going on. Here is a copy of those notes for interest.
I used package managers to provide PostgreSQL support as that was what the Docker install gave me to start with. I may have chosen another DB engine but as I was already using Postgres I stuck with that. I also installed Xapian using the package manager as well as Exim, Nginx, Sassc and Python using the package managers. I didn't install Memcached.
I mainly followed the instructions at: https://wiki.list.org/DOC/Mailman%203%20installation%20experience
My directory structure uses /opt/mailman as the base, the instructions on this web page use /opt/mailman/mm and I didn't have this extra subdirectory. I ensured this directory was owned by list:list.
I installed my venv under /opt/mailman/venv choosing to copy over the existing distribution installed Python packages and installed the following packages via Pip after activating it:
Upgrade packages that are already installed which need an upgrade to work with Mailman3:
pip3 install -U pip zope.interface
Get the rest installed:
pip3 install mailman mailmanclient django-mailman3 mailman-hyperkitty hyperkitty postorius psycopg2-binary xapian-haystack
This also installs Gunicorn which I use as the backend wsgi server.
The reason I chose to use existing system packages as part of the venv is because I wanted to use the Xapian installation I had installed via system packages as I ran into issues installing it via Pip. In future I think I will try and install the PostgreSQL driver this way as well.
Once deactivating the venv I recommend getting the directory structure and files copied to the right places from the links on the wiki, noting that in my cases I had to edit the files to change the relevant paths to suit my new layout, and also where necessary (Systemd units) change the user/group. . You should then run all the Mailman specific commands using these scripts in /opt/mailman/bin. I learnt the hard way that you will run into problems if you try and run Mailman from the venv directory as it will look for files in the wrong place and create data files in the wrong directory. As I went down this road my configuration files mailman.cfg and mailman-hyperkitty.cfg are actually located in /opt/mailman/etc but that isn't a requirement if you follow the wiki and use the scripts provided.
I obtained manage.py, settings.py, urls.py and wsgi.py from the Gitlab Mailman-suite project instead of the wiki site to ensure I got the latest versions and put these files in /opt/mailman. Regarding /opt/mailman/bin/mailman-post/update, I removed the references to Memcached as I didn't use it. I also didn't symlink /opt/mailman/logs to /opt/mailman/var/logs as advised by the wiki but will do this on a reinstall.
I generated a new Mailman PostgreSQL user and restored my existing database to the new server (which also upgraded the DB to a newer Postgresql version as the Docker-Compose from the Mailman Docker project uses an older Postgresql container).
I copied the Mailman runtime files from the Mailman core container data volume to /opt/mailman/var.
I generated a Hyperkitty API secret key and also a Django secret key.
Taking the mailman.cfg file from the wiki as a base, I updated the following parts:
[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://mailman:[password]@localhost/mailmandb
[mailman]
site_owner: mailman-owner(a)lists.domain.com
[mta]
incoming: mailman.mta.exim4.LMTP
configuration: python:mailman.config.exim4
[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /opt/mailman/etc/mailman-hyperkitty.cfg
In /opt/mailman/etc/mailman-hyperkitty.cfg add the following:
[general]
base_url: http://localhost:8000/hyperkitty/
api_key: [generated-hyperkitty-api-key]
Using systemctl see if you can start Mailman and observe logs etc.
In terms of the Django stuff you need to create /opt/mailman/settings-local.py which will be read in by the system and will overwrite the default settings in /opt/mailman/settings.py obtained from the Mailman-suite project. This directly configures the Django framework which the Mailman web components Postorius and Hyperkitty run under.
Here is a outline copy of my file:
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
SECRET_KEY = '[generated-django-secret-key]'
MAILMAN_ARCHIVER_KEY = '[generated-hyperkitty-api-key]'
DEFAULT_FROM_EMAIL = 'mailman-owner(a)lists.domain.com'
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
DEBUG = False
USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
ALLOWED_HOSTS = [
'localhost',
'lists.domain.com'
]
INSTALLED_APPS = (
# Copy the installed apps from the settings.py and remove the social provider logins you don't want to support.
)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'mailmandb',
'USER': 'mailman',
'PASSWORD': '[password]',
'HOST': 'localhost'
}
}
HAYSTACK_CONNECTIONS = {
'default': {
'PATH': os.path.join(BASE_DIR, "fulltext_index"),
'ENGINE': 'xapian_backend.XapianEngine'
},
}
LOGGING = {
# Copy the complete LOGGING section from settings.py and change the path as follows, if there is a way of just adding the relevant path override to this file let me know. I am doing this only because I didn't create a symlink for /opt/mailman/logs.
'version': 1,
'disable_existing_loggers': False,
'filters': {
[...]
}
},
'handlers': {
'mail_admins': {
[...]
},
'file':{
'level': 'INFO',
'class': 'logging.handlers.WatchedFileHandler',
'filename': os.path.join(BASE_DIR, 'var/logs', 'mailmansuite.log'),
'formatter': 'verbose',
},
'console': {
[...]
},
},
'loggers': {
'django.request': {
[...]
},
'django': {
[...]
},
'hyperkitty': {
[...]
},
'postorius': {
[...]
},
},
'formatters': {
'verbose': {
[...]
},
'simple': {
[...]
},
},
}
Try running /opt/mailman/bin/mailman-post-update and choose to rebuild indexes, see if the static files get saved to /opt/mailman/static and the search indexes get saved to /opt/mailman/fulltext_index. In my case I had plenty of indexing to do, not sure if these files are created on a green-field installation with nothing archived. This command will also populate the Postgresql database schema.
See if the Systemd units for mailman-web and mailman-cluster come up. Observe logs. If everything ok the Gunicorn should be listening on localhost:8000.
For Nginx integration I followed the wiki, only change I had to make was to send the host header to the backend server, here is a partial config file:
server {
server_name lists.domain.com;
root /var/www/lists;
access_log /var/log/nginx/lists-access.log;
error_log /var/log/nginx/lists-error.log warn;
listen [::]:443 ssl; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/lists.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/lists.domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location = /favicon.ico {
log_not_found off;
}
location = /robots.txt {
log_not_found off;
}
location / {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_pass http://localhost:8000/;
}
location /static {
alias /opt/mailman/static;
}
}
server {
if ($host = lists.domain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80;
server_name lists.domain.com;
return 404; # managed by Certbot
}
For Exim integration I followed to the letter the information here: https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#exim
Here is a Logrotate script which is not in the Mailman wiki:
/opt/mailman/var/logs/*.log {
missingok
sharedscripts
su list list
postrotate
/opt/mailman/bin/mailman reopen &>/dev/null || true
if [ -r /opt/mailman/var/gunicorn.pid ]; then
PID=`cat /opt/mailman/var/gunicorn.pid`
kill -s USR1 $PID
fi
endscript
}
I have been running this setup since March 2020, I run around 10 lists on this server with around 10 mails per list per day. I actually had more issues when moving from Mailman2 to Mailman3 using the Docker containers and actually this process made me understand a lot more how everything hangs together and where to troubleshoot issues.
Hope this helps someone out there.
Andrew.
-----Original Message-----
From: Matthew Alberti <matthew(a)alberti.us>
Sent: 01 January 2021 02:48
To: ieso(a)johnwillson.com
Cc: 'Mailman users' <mailman-users(a)mailman3.org>
Subject: [MM3-users] Re: Easiest way to install a new mailman3 deployment?
I had a very similar experience. I don't mean to knock the dev team; Mailman3 is an awesome tool that really meets a need. It is really much much more than a 'tool.' A very comprehensive software suite.
For me, I ended up using the docker option on Ubuntu 20.04. My advice is to give up on the idea of using OS package managers. They just aren't current enough (certainly for Ubuntu LTS), or updated frequently enough, for the current development of Mailman3. Nothing but problems for me on Ubuntu, and Debian is about the same.
Pypi, or installation in a virtualized environment, was equally challenging. I strongly recommend the docker installation method. It is not quite turnkey, but the documentation is close enough where it might 'just work' in your environment. I used MariaDB, and that introduced some challenges. If you are using postgre, postfix, and nginx, you are pretty close to a very easy installation.
This list is very supportive, with very frequent responses from the core dev team. And there are guys like me who really struggled to get to a production instance. Please let us all know what isn't working in the docker approach, and we will try to help I'm sure.
- Matt Alberti
Get BlueMail for Android
On Dec 31, 2020, 8:15 PM, at 8:15 PM, ieso(a)johnwillson.com wrote:
>So I've been trying for the last two weeks to get a new mailman3 server
>running on a virtualized server (any server), and I'm turning to this
>list after having failed many times and running out of holiday time.
>
>I started trying a non-docker installation on Ubuntu 18.04
>(https://docs.google.com/document/d/1xIcSsoNFp2nHi7r4eQys00s9a0k2sHhu1V
>5PlantPTs/edit#) , which got me the closest. Except I had a problem
>with inbound email only being triggered when it came from certain
>accounts. But that clearly wasn't good enough for production, so after
>many attempts to figure out where it was failing, I decided to turn to
>docker as a solution that should be cleaner.
>
>A few attempts at doing a docker installation on digitalocean.com
>failed, which I realized might be due to it not routing private IP
>addresses, so I moved to AWS after checking that their VPC policy would
>fit mailman's docker requirements. I found a great but slightly
>outdated guide on how to do this
>(https://xiaoxing.us/2018/01/01/deploy-mailman-3-on-aws-using-docker/)
>By this point I knew enough to correct a number of places where the
>environment had changed since the procedure was written, but postorius
>still failed at the curl test.
>
>The challenge for me has been the difficulty to know how to
>troubleshoot the different different systems and network infrastructure
>that are used to keep mailman3 humming. I've tried about 7 different
>installation walkthroughs (there are no recent ones on Youtube by the
>way, in case anyone wants to seize that opportunity!), and the good
>guides provide ways to check each stage to try to help you a bit on
>that front.
>
>Nonetheless, I feel stuck and thought I'd ask the simple question...
>for a completely basic, barebones new installation, what's the easiest
>way to get a mailman3 installation up-and running? (e.g. Which server
>provider? Which operating system and version? Docker or otherwise?)
>
>Any pointers highly appreciated. Google Groups is clearly on its way
>out, as it no longer allows for people to easily join groups by sending
>an email or clicking a link, so that should be a big opportunity for
>mailman3 to step up and help give those mailing list migrants a new
>home... which is what we're looking for. We're just not quite as smart
>as you guys. ;-) _______________________________________________
>Mailman-users mailing list -- mailman-users(a)mailman3.org To unsubscribe
>send an email to mailman-users-leave(a)mailman3.org
>https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org To unsubscribe send an email to mailman-users-leave(a)mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
4 years, 6 months
Can login at /admin, but cannot login at the account/login page.
by pabsta＠gmail.com
Hi All,
thanks for your contibution to opensource softwares. I have done the installation walkthrough on the wiki page (thanks to Brian Carpenter) and I am almost there. I can run the three services and they provide the relevant web interfaces (admin, postorius, hyperkitty). In comparison to the walkthrough, I did two changes; First, I used different port numbers , as it would otherwise conflict with other services on the server. Second, I disabled the social accounts authentification mechanisms. I am still running on sqllite, as I am still testing things (one task at a time).
Using my "superuser" credentials, I can log to the admin section in a fraction of a second. The server response time is less than one second. Once logged in, I can navigate through all the site as an authenticated user, including in the postorius pages related to list management. The services work normally.
However, if logout and attempt to login from the postorius website, I get a « 502 bad gateway » error after roughly 30 seconds. The logs in Systemctl for gunicorn register a "DIGEST-MD5 common mech free" error message and the nginx error logs just show a connection drop:
2021/08/12 18:11:09 [error] 26026#26026: *31 upstream prematurely closed connection while reading response header from upstream, client: [ip redacted], server:
[domain redacted], request: "POST /accounts/login/ HTTP/2.0", upstream: "http://127.0.0.1:8006/accounts/login/", host: "[domain redacted]", referrer: "https://[domain redacted]/accounts/login/?next=/mailman3/lists/"
Two additional clues may help understanding what is going on:
1- If I attempt to login with improper credentials, the services respond normally, and I get the expected red box with "invalid credentials" message on the postorius page. It is only when proper credentials are given that the connection fails.
2- If I attempt to register a new email adress, the services also fail to respond (although with no DIGEST-MD5 message registered).
It thus seems to be a post "POST" problem, but what puzzles me is that it is not an issue for the admin page.
Below the relevant sections of my settings.py page. The skipped ([...]) sections are as in the walkthrough files. In particular, the AUTH_PASSWORD_VALIDATORS, the EMAIL_BACKEND are as is and the "USE_X_FORWARDED_HOST" is still commented.
##Settings.py
[...]
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '[redacted]'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = False
ADMINS = (
('ME', 'my@email'),
)
SITE_ID = 1
# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.8/ref/settings/#allowed-hosts
ALLOWED_HOSTS = [
"localhost", # Archiving API from Mailman, keep it.
"[domain]",
# Add here all production URLs you may have.
]
# Mailman API credentials
MAILMAN_REST_API_URL = 'http://localhost:8005'
MAILMAN_REST_API_USER = '[redacted]'
MAILMAN_REST_API_PASS = '[redacted]'
MAILMAN_ARCHIVER_KEY = '[redacted]'
MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1')
# Application definition
INSTALLED_APPS = (
'hyperkitty',
'postorius',
'django_mailman3',
# Uncomment the next line to enable the admin:
'django.contrib.admin',
# Uncomment the next line to enable admin documentation:
# 'django.contrib.admindocs',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'django_gravatar',
'compressor',
'haystack',
'django_extensions',
'django_q',
'allauth',
'allauth.account',
'allauth.socialaccount',
#'django_mailman3.lib.auth.fedora',
#'allauth.socialaccount.providers.openid',
#'allauth.socialaccount.providers.github',
#'allauth.socialaccount.providers.gitlab',
#'allauth.socialaccount.providers.google',
#'allauth.socialaccount.providers.facebook',
#'allauth.socialaccount.providers.twitter',
#'allauth.socialaccount.providers.stackexchange',
)
[...]
# Database
# https://docs.djangoproject.com/en/1.8/ref/settings/#databases
DATABASES = {
'default': {
# Use 'sqlite3', 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
'ENGINE': 'django.db.backends.sqlite3',
# DB name or path to database file if using sqlite3.
'NAME': os.path.join(BASE_DIR, 'mailmansuite.db'),
# The following settings are not used with sqlite3:
'USER': 'XXX',
'PASSWORD': 'XXX',
# HOST: empty for localhost through domain sockets or '127.0.0.1' for
# localhost through TCP.
'HOST': '',
# PORT: set to empty string for default.
'PORT': '',
# OPTIONS: for mysql engine only, do not use with other engines.
# 'OPTIONS': {'charset': 'utf8mb4'} # Enable utf8 4-byte encodings.
}
}
[...]
# Password validation
# https://docs.djangoproject.com/en/1.9/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME':
'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME':
'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME':
'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME':
'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/1.8/topics/i18n/
LANGUAGE_CODE = 'fr'
TIME_ZONE = 'America/New_York'
[...]
# If you enable internal authentication, this is the address that the emails
# will appear to be coming from. Make sure you set a valid domain name,
# otherwise the emails may get rejected.
# https://docs.djangoproject.com/en/1.8/ref/settings/#default-from-email
# DEFAULT_FROM_EMAIL = "mailing-lists(a)you-domain.org"
DEFAULT_FROM_EMAIL = 'username@domain'
# If you enable email reporting for error messages, this is where those emails
# will appear to be coming from. Make sure you set a valid domain name,
# otherwise the emails may get rejected.
# https://docs.djangoproject.com/en/1.8/ref/settings/#std:setting-SERVER_EMAIL
# SERVER_EMAIL = 'root(a)your-domain.org'
SERVER_EMAIL = 'username@domain'
# Change this when you have a real email backend
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
[...]
#
# Social auth
#
AUTHENTICATION_BACKENDS = (
'django.contrib.auth.backends.ModelBackend',
'allauth.account.auth_backends.AuthenticationBackend',
)
# Django Allauth
ACCOUNT_AUTHENTICATION_METHOD = "username_email"
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_EMAIL_VERIFICATION = "mandatory"
# You probably want https in production, but this is a dev setup file
ACCOUNT_DEFAULT_HTTP_PROTOCOL = "https"
ACCOUNT_UNIQUE_EMAIL = True
#
# Asynchronous tasks
#
Q_CLUSTER = {
'timeout': 300,
'save_limit': 100,
'retry':360,
'orm': 'default',
}
[...]
# Only display mailing-lists from the same virtual host as the webserver
FILTER_VHOST = True
POSTORIUS_TEMPLATE_BASE_URL = 'http://localhost:8006'
## (End of settings.py)
I'm happy to provide additional information if it can help.
Best,
Pier-André
3 years, 10 months
Re: Timeouts
by Abhilash Raj
> On Apr 25, 2021, at 7:06 PM, tlhackque via Mailman-users <mailman-users(a)mailman3.org> wrote:
>
> On 25-Apr-21 20:34, Mark Sapiro wrote:
>> On 4/25/21 4:37 PM, tlhackque via Mailman-users wrote:
>>
>>> The described timeouts are something that hyperkitty ought to be able to
>>> avoid. For apache, the timeout is idle time between blocks of output.
>>> Hyperkitty can avoid this by generating the archive in segments (based
>>> on size, or elapsed time), flushing its output buffer, generating a
>>> multi-file archive, and/or using Transfer-Encoding: chunked (chunked
>>> doesn't work for http/2). It ought to be able to break the
> work into
>>> blocks of "n" messages & do something to generate output. Besides
>>> avoiding timeouts, working in segments allows the GUI to display
>>> meaningful progress (e.g. if you're loading with XMLHttpRequest,
>>> "onprogress") It really oughtn't be up to the user to break up the
>>> request.
>> It is not the web server that times out. I'm not sure about uwsgi
>> because I don't use it, but the timeouts I see are on servers that use
>> gunicorn as the WSGI interface to Django and the timeout is in a
>> gunicorn worker. This is controlled by the timeout setting in the
>> gunicorn config. <https://docs.gunicorn.org/en/stable/settings.html#timeout>
>>
>> Note that even 300 seconds is not enough to download the entire
>> <https://mail.python.org/archives/list/python-dev@python.org/> archive.
>>
>> It may be possible to get HyperKitty to chunk the output to avoid this,
>> but it doesn't currently do that. Care to submit an MR?
>>
>>
> I'm afraid (u)WSGI, Django, and gunicorn are not technologies that I
> work with.
>
> It sounds as if hyperkitty is compiling the entire archive before
> sending the first byte.
>
> The gunicorn doc that you pointed to says
>
> Workers silent for more than this many seconds are killed and restarted.
> Setting it to 0 has the effect of infinite timeouts by disabling
> timeouts for all workers entirely.
>
> "Silent" sounds like the standard webserver "you have to push some bits,
> or we assume you're stuck".
>
> My understanding is that gunicorn is a Python persistence server that is
> run behind a webserver proxy. So the (proxy) webserver (apache, nginx,
> ...) timeouts also apply and would need to be increased.
>
> Might be interesting to try 0 (gunicorn) / 1200 (webserver) with your
> python-dev archive, time it and see how much (encoded) data is
> transferred... (I would expect most mailing list archives to compress
> nicely, though those with binary attachments wont.)
For uwsgi, I think the parameter is called `harakiri`[1][2] (I don’t know why such a name though).
[1]: https://uwsgi-docs.readthedocs.io/en/latest/Options.html#harakiri
[2]: https://uwsgi-docs.readthedocs.io/en/latest/Glossary.html
> if request takes longer than specified harakiri time (in seconds), the request will be dropped and the corresponding worker recycled.
This should be set to a long enough value that allows downloading the archive.
If you are using http socket, then you want http-timeout.
Also, to set the timeout in webserver (nginx)
location / {
uwsgi_read_timeout 120s;
uwsgi_send_timeout 120s;
uwsgi_pass 0.0.0.0:8000;
include uwsgi_params;
}
Or some other value that you want.
> But fiddling with timeouts is treating the symptom, not the root cause.
> The right solution is to stream, segment (or chunk) the output, because
> in the general case, no timeout is long enough. It'll always be
> possible to find an archive that's just one byte (or second) longer than
> any chosen timeout. (See the halting problem.) You want
> the timeout
> to catch a lack of progress, not total time that's a function of
> transaction size. (Webservers may also have limits on transaction size
> - e.g. mod_security, but they're only useful when the upper bound on a
> response is knowable.) Thus, the timeout(s) should be roughly
> independent of the archive size; on the order of time-to-first-byte
> (which ordinarily is longer than time between segments/chunks).
>
> Also note that streaming requires fewer server resources than compiling
> a complete archive before sending, since you don't need to create the
> entire archive in memory (or in a tempfile). You only need enough
> memory to efficiently buffer the file I/O and to contain the compression
> tables/output buffer. Except for trivial cases, this will be
> independent of the archive size. The only downside is that if the comm
> link is slow, you may hold a reader lock on the source data for longer
> than necessary with the current scheme.
>
> Seems as though this deserves an issue. I guess I could open one - but
> you have the experience/test cases.
Hyperkitty doesn’t actually create an archive in memory or in a temp file. It uses streaming response with on the fly compression to read from database and relay to the client for download.
https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/views/mlist.…
The problem could be that uwsgi seems to kill an ongoing downloading process, not an idle one for some reason. And, it seems that it is a known and intentional behavior. I don’t see a good way to disable it completely, but perhaps it can be set to a long enough value that it never essentially kills a running worker which is moving bits.
--
thanks,
Abhilash Raj (maxking)
4 years, 2 months
Re: How To Install Mailman 3 on Debian 10 (Complete Guide)
by Abhilash Raj
On Thu, Feb 25, 2021, at 6:38 PM, Brian Carpenter wrote:
> On 2/25/21 8:28 PM, Abhilash Raj wrote:
> > Thanks Brian, this looks quite comprehensive in the details. The only
> > thing I am a bit concerned about is granting sudo privileges to the
> > Mailman user. It really shouldn't have sudo given that Mailman and
> > Django are supposed to run as mailman user. Any compromise of
> > the Django application will provide the attacker root on the machine.
>
> Thanks for pointing that out. I have removed that step. It really wasn't
> needed.
Great, thanks!
>
> > Is there a specific reason that you chose to go with an entirely new doc
>
> Yes. Just from observing the communications on this list, there are
> multiple ways to install Mailman 3 which produces installation issues,
> database problems, outdated apps, etc. My document produces a VERY
> stable installation of Mailman 3 and one that is easy to maintain and
> update. It is also very comprehensive covering not just Mailman Suite,
> but the configuration of an entire server environment. Personally I
> think the choice of Debian, NGINX, Postfix, and PostgreSQL makes for a
> long and loving relationship with Mailman 3 but I do understand everyone
> has their own environment that they want to work with. My guide
> represents a very stable and easily maintained environment for me and
> hopefully others.
I think it is okay to be pick up a specific configuration and provide a
guide to installation keeping that as the target technology. As it happens
to be, the official doc uses the same set, Nginx, Postfix and PostgSQL.
https://docs.mailman3.org/en/latest/install/virtualenv.html
I don't see anything very specific to Debian in the doc except the package
names and I think they are similar enough in Debian derivative distros that
we can write up that assumption up front in the top.
Given that most of the installation is based off on pip, I don't think there is
going to be a *lot* of deviation for even RHEL/Fedora based distros when it
comes to Mailman itself. Some minute deviations could occur with the
behavior of certain commands, like `nano` doesn't exist, but I still expect
95% of the doc to be the same, except obviously the package names.
>
> > rather than helping to improve the existing one? Several parts of it (at
> > least the ones that official guide covers) seems similar to me and is
> > duplicate information that at least two people are going to spend time
> > writing and maintaining in future.
>
> Is there an official guide? See that is the problem. I did not know
> there was an official guide. Because every time I tried finding
> documentation in the past using Google searches, I got fed outdated
> information. So for me the guide that I came up with, works great for me
> and if no one else uses it, I certainly will.
I am not that well versed in SEO, but the official website https://list.org
does point to it. Maybe not prominently enough for new users to discover?
I'll see if we can make the discovery of docs.mailman3.org better.
>
> Your guide, by nature has to be minimalistic to accommodate all the
> various server environments out there. My is meant to be narrow but
> comprehensive.
I guess it is fine to be opinionated about choosing the technologies but
yet provide pointers to help people using other technologies.
Right now, I do that using "Hints" and "Notes" and "See also" banners
through out the documentation to point to different pages where they
can lookup setup for other technologies. For example, how to setup
MySQL if official doc recommends Postgres.
>
> > Is it something about the contribution process to the official documentation
> > that makes it hard for people to contribute? Most of the pages at
> > docs.mailman3.org or come from this[1] repo and use Sphinx to build and
> > REsT formatting (.rst).
> >
> > [1]:https://gitlab.com/mailman/mailman-suite-doc
>
> Well I don't know much about any of the above approach to documentation.
> So I would say that is a hindrance.
Gitlab has a fairly good experience editing REsT files and you don't necessarily
have to do much outside of a Web Browser actually. It lets you preview before
saving like wiki.list.org and you can submit it for review and merging into
main repo.
Would you be more willing to contribute to official documentation if we
documented the Gitlab contribution process better?
> > I am just trying to understand how can we lower the barrier for community
> > members to help contribute to existing docs instead of them having to create
> > new ones. Specifically around installation, since that tends to get stale often
> > when depedent packages change or a new dependency is added that
> > breaks the installation.
>
> I had to update my guide twice due to recent changes made to the
> installation process such as the addition of the Dart Sass section and
> the problem with the cryptography module when installing Hyperkitty.
> While my guide covers these new hurdles to a new installation, I don't
> think yours says anything about them. The Dart Sass section came from
> Mark's comments on this list.
Because we haven't yet gotten to updating those in the official docs,
unfortunately. And there are more things missing like Fulltext search (Xapian
recommended) and more of the Email setup like SPF and DKIM. Very much
aligned with your plans, which is why I suggested contributing updates
to docs.mailman3.org instead :)
--
thanks,
Abhilash Raj (maxking)
4 years, 4 months
Re: Mixing Mailman 3 with AWS SES
by Ralph Weber
To close the loop on this topic...
Comments received directly from one of the people who developed the AWS
Marketplace kits for Mailman 3 and Mailman 4 have definitively
established that AWS Mailman is *not* intended to interact with AWS SES
when receiving email messages.
AWS Mailman is designed to receive its email messages via Postfix.
With many thanks to all those on this list who corroborated "Mailman and
SES don't mix" mantra here, we are taking the advice received from so
many sources and ending efforts to do the impossible.
All the best,
.Ralph
On 3/20/2021 8:23 AM, Stephen J. Turnbull wrote:
> Ralph Weber writes:
>
> > I am moving the website www.t10.org from a private/corporate VM to
> > AWS EC2s.
> >
> > For many years, t10.org has had very happy results with using
> > Mailman 2 for its email reflector, and I have no real concerns
> > about moving to Mailman 3. My bugbear is the fact that I need the
> > following mixed bag of incoming email addresses to work in concert
> > with each other to receive and handle emails.
> >
> > > t10(a)t10.org (Primary Mailman Reflector)
> > > chair(a)t10.org (Mailman Reflector to direct messages to the T10
> > > chair and his designates)
> > * docs(a)t10.org (document posting mechanics - serviced by a small
> > mountain of home-grown code code
> > * bbs(a)t10.org (other publicly accessible services - serviced by
> > code that recalls the days when everything as done on a Bulletin
> > Board System, BBS)
>
> This isn't a Mailman question, to be frank. As Abhilash points out,
> Mailman doesn't care about other mailboxes served by its host (unless
> they conflict with list names or adinistrative addresses derived from
> list names, which evidently yours do not). It's a question about the
> mail service at AWS. I've not seen any documentation in our project
> about setting up at AWS; I don't know anybody who has done so. (I
> haven't looked for either though.)
>
> I took a quick look for AWS SES documentation. They don't seem to
> have any real documentation online for the general public, only lists
> of features. They also seem to be very focused on sending rather than
> receiving. Much talk of how much mail you can send, bulk emailing,
> statistics on responses, and the like. None about setting up for
> incoming mail.
>
> Probably I could get access to online help if I signed up for a free
> account, but that's above my pay grade.
>
> > My best guess is that the AWS Simple Email Service (SES) needs to
> > sit in front of both Mailman and the home-grown code,
>
> I don't know if AWS provides an alternative Complicated Email Service,
> but yes, something like SES is going to have to sit in front of the
> various mailboxes.
>
> Mailman itself has a certain amount of capability to deal with the
> main email protocols, but it is definitely not capable of replacing
> the main mail server. For security reasons I would definitely advise
> against any Mailman 3 presence directly on the Internet. It was
> designed on the assumption that it would talk to the Internet via a
> standard mail server, Postorius (for administration), and HyperKitty
> (for mail archives)[1], and occasionally to admins logging in with shell
> accounts.
>
> > All available evidence read to date
>
> URLs, please. As I wrote above, Amazon's documentation is at best not
> very discoverable.
>
> > suggests that the default AWS installation
>
> Whose default installation of Mailman? AWS's? Typical 3rd party
> configurations?
>
> > of Mailman 3 assumes that Mailman 3 is the *only* receiver of
> > emails in the configuration.
>
> That's possible, but it's likely an artifact of the application -- I
> would expect that many who host Mailman on AWS do so to isolate
> Mailman from other services they provide. So it seems reasonable that
> in many such installations Mailman is *the* service for that instance.
>
> But mail servers are designed to be many inputs, many outputs
> services. If SES itself is as restricted as you fear, surely AWS
> provides an alternative, more powerful mail service. I can imagine
> quite a few restrictions, other than the backing application, on the
> SES that Amazon *might* impose. But without access to documentation,
> we can't say much.
>
> > Or... Hercules may have already cleaned the Augean Stables, and
> > left behind a AWS Lambda function that can serve as a keystone
> > puzzle piece for solving my dilemma.
>
> Good luck on that. I don't know of any, but that would be a quick
> solution for you. I hope somebody here knows more. As evidenced by
> Andrew's post, there are folks thinking about it, so maybe somebody's
> actually tried -- anybody? Successes? Pitfalls to watch for?
>
> The other issue Andrew mentions could be important. Since Solar Winds
> I'm seeing daily attempts to access my O365 server (nonexistent) from
> AWS. Given the size of AWS, I think they probably actually do a good
> job of keeping felonious behavior to a minimum[2], but a lot of folks
> I know are happy to cut off huge netblocks until somebody they know
> actually complains. :-(
>
> Footnotes:
> [1] Or more precisely, some webservice providing those services
> mediating between the Internet and Mailman sitting cozily behind a
> firewall.
>
> [2] DigitalOcean, on the other hand, is accumulating blocked /24s,
> and even /16s, at my firewall. But that's another story.
4 years, 3 months