Re: Hyperkitty CPU usage
by Alain Kohli
I have not seen any improvement over time, so I switched the backend to
Xapian. That works like a charm now, I guess our lists were indeed just
too busy for whoosh.
On 5/7/19 5:15 PM, Alain Kohli wrote:
> Well, I understand that it might take a while, but the process is
> running for over 200h now, I doubt it can really take _that_ long if
> everything is working properly. Especially because the rebuild_index
> command is only taking a few hours at most, doesn't that essentially
> do the same?
>
> We have ~700k messages in ~300 mailing lists in case that helps you
> estimate how long it could take. I can take a look at alternative
> backends, though.
>
>
> On 5/7/19 4:32 PM, Abhilash Raj wrote:
>> On Sat, Apr 27, 2019, at 6:22 PM, Alain Kohli wrote:
>>> I'm running a custom image which is based on an older version of the
>>> one
>>> here: https://github.com/maxking/docker-mailman. I attached it below.
>>> But I separated postorius and hyperkitty, so hyperkitty is running in
>>> its own container. I'm deploying the image with a plain 'docker run'
>>> behind nginx. I made fulltext_index persistent now, but it didn't get
>>> populated with anything yet. I don't really have an error traceback
>>> because there is never an error thrown. The only thing with some
>>> content
>>> is uwsgi-error.log, which you can find below. I'm also still getting
>>> the
>>> "A string literal cannot contain NUL (0x00) characters." messages. I
>>> also noticed that it takes incredibly long for the webinterface to load
>>> (several minutes) even though there doesn't seem to be any process
>>> consuming notable resources apart from the minutely job.
>>>
>>> Funnily enough, I have the exact same image deployed on a second server
>>> as well for testing. On that one everything works fine. The only
>>> difference is that on the problematic one I have a lot more mailing
>>> lists/archives and that I imported them from mailman2. Could something
>>> have gone wrong during the import? I used the regular hyperkitty_import
>>> command.
>> Yes, this is because `whoosh`, the library set by default to run
>> fulltext
>> indexing is a pure python implementation and quite slow in busy lists.
>>
>> We do support more backends though, see [1] for a list of all the
>> supported
>> search backends. Something like Xapian(C++) or Elasticsearch/Solr(Java)
>> should be much better in terms of performance.
>>
>> [1]:
>> https://django-haystack.readthedocs.io/en/master/backend_support.html
>>
>>> uwsgi-error.log:
>>>
>>> *** Starting uWSGI 2.0.18 (64bit) on [Sat Apr 27 22:50:17 2019]
>>> ***
>>> compiled with version: 6.4.0 on 27 April 2019 22:48:42
>>> os: Linux-4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19)
>>> nodename: hyperkitty.docker
>>> machine: x86_64
>>> clock source: unix
>>> detected number of CPU cores: 4
>>> current working directory: /home/hyperkitty
>>> detected binary path: /usr/local/bin/uwsgi
>>> !!! no internal routing support, rebuild with pcre support !!!
>>> setgid() to 82
>>> setuid() to 82
>>> chdir() to /home/hyperkitty
>>> your memory page size is 4096 bytes
>>> detected max file descriptor number: 1048576
>>> lock engine: pthread robust mutexes
>>> thunder lock: disabled (you can enable it with --thunder-lock)
>>> uwsgi socket 0 bound to TCP address 0.0.0.0:8081 fd 8
>>> uwsgi socket 1 bound to TCP address 0.0.0.0:8080 fd 9
>>> Python version: 3.6.8 (default, Jan 30 2019, 23:54:38) [GCC 6.4.0]
>>> Python main interpreter initialized at 0x55dfaa41c980
>>> python threads support enabled
>>> your server socket listen backlog is limited to 100 connections
>>> your mercy for graceful operations on workers is 60 seconds
>>> [uwsgi-cron] command "./manage.py runjobs minutely" registered as
>>> cron task
>>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly"
>>> registered
>>> as cron task
>>> [uwsgi-cron] command "./manage.py runjobs hourly" registered as
>>> cron
>>> task
>>> [uwsgi-cron] command "./manage.py runjobs daily" registered as
>>> cron task
>>> [uwsgi-cron] command "./manage.py runjobs monthly" registered as
>>> cron task
>>> [uwsgi-cron] command "./manage.py runjobs weekly" registered as
>>> cron
>>> task
>>> [uwsgi-cron] command "./manage.py runjobs yearly" registered as
>>> cron
>>> task
>>> mapped 208576 bytes (203 KB) for 4 cores
>>> *** Operational MODE: threaded ***
>>> WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter
>>> 0x55dfaa41c980 pid: 1 (default app)
>>> *** uWSGI is running in multiple interpreter mode ***
>>> spawned uWSGI master process (pid: 1)
>>> spawned uWSGI worker 1 (pid: 40, cores: 4)
>>> Sat Apr 27 22:50:18 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> minutely" (pid 45)
>>> [uwsgi-daemons] spawning "./manage.py qcluster" (uid: 82 gid: 82)
>>> 22:50:21 [Q] INFO Q Cluster-47 starting.
>>> 22:50:21 [Q] INFO Process-1:1 ready for work at 59
>>> 22:50:21 [Q] INFO Process-1:2 ready for work at 60
>>> 22:50:21 [Q] INFO Process-1:3 ready for work at 61
>>> 22:50:21 [Q] INFO Process-1:4 ready for work at 62
>>> 22:50:21 [Q] INFO Process-1:5 monitoring at 63
>>> 22:50:21 [Q] INFO Process-1 guarding cluster at 58
>>> 22:50:21 [Q] INFO Process-1:6 pushing tasks at 64
>>> 22:50:21 [Q] INFO Q Cluster-47 running.
>>> 22:59:31 [Q] INFO Enqueued 3403
>>> 22:59:31 [Q] INFO Process-1:1 processing [update_from_mailman]
>>> 22:59:33 [Q] INFO Processed [update_from_mailman]
>>> Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> quarter_hourly" (pid 73)
>>> Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> hourly" (pid 74)
>>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
>>> with pid 73 exited after 64 second(s)
>>> 23:01:28 [Q] INFO Enqueued 3404
>>> 23:01:29 [Q] INFO Process-1:2 processing
>>> [rebuild_mailinglist_cache_recent]
>>> [uwsgi-cron] command "./manage.py runjobs hourly" running with pid
>>> 74 exited after 91 second(s)
>>> Sat Apr 27 23:01:36 2019 - uwsgi_response_write_body_do(): Broken
>>> pipe [core/writer.c line 341] during GET / (212.203.58.154)
>>> OSError: write error
>>> 23:01:36 [Q] INFO Processed [rebuild_mailinglist_cache_recent]
>>> Sat Apr 27 23:15:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> quarter_hourly" (pid 88)
>>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
>>> with pid 88 exited after 4 second(s)
>>> 23:28:24 [Q] INFO Enqueued 3405
>>> 23:28:24 [Q] INFO Process-1:3 processing [update_from_mailman]
>>> 23:28:25 [Q] INFO Processed [update_from_mailman]
>>> Sat Apr 27 23:30:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> quarter_hourly" (pid 96)
>>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
>>> with pid 96 exited after 4 second(s)
>>> 23:44:40 [Q] INFO Enqueued 3406
>>> 23:44:40 [Q] INFO Process-1:4 processing [update_from_mailman]
>>> 23:44:41 [Q] INFO Processed [update_from_mailman]
>>> Sat Apr 27 23:45:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> quarter_hourly" (pid 104)
>>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
>>> with pid 104 exited after 4 second(s)
>>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> quarter_hourly" (pid 113)
>>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> hourly" (pid 114)
>>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> daily" (pid 115)
>>> Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py
>>> runjobs
>>> weekly" (pid 116)
>>> [uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
>>> with pid 113 exited after 55 second(s)
>>> [uwsgi-cron] command "./manage.py runjobs weekly" running with pid
>>> 116 exited after 55 second(s)
>>> 00:01:36 [Q] INFO Enqueued 3407
>>> 00:01:36 [Q] INFO Process-1:1 processing
>>> [rebuild_mailinglist_cache_recent]
>>> [uwsgi-cron] command "./manage.py runjobs hourly" running with pid
>>> 114 exited after 99 second(s)
>>> 00:01:50 [Q] INFO Processed [rebuild_mailinglist_cache_recent]
>>> 00:04:52 [Q] INFO Enqueued 3408
>>> 00:04:52 [Q] INFO Process-1:2 processing [update_from_mailman]
>>> 00:04:54 [Q] INFO Processed [update_from_mailman]
>>>
>>> Dockerfile:
>>>
>>> FROM python:3.6-alpine3.7 # Add startup script to container COPY
>>> assets/docker-entrypoint.sh /usr/local/bin/ # Install packages and
>>> dependencies for hyperkitty and add user for executing apps. #
>>> It's
>>> important that the user has the UID/GID 82 so nginx can access the
>>> files. RUN set -ex \&& apk add --no-cache --virtual .build-deps
>>> gcc
>>> libc-dev linux-headers git \postgresql-dev \&& apk add --no-cache
>>> --virtual .mailman-rundeps bash sassc mailcap \postgresql-client
>>> curl \&& pip install -U django==2.2 \&& pip install
>>> git+https://gitlab.com/eestec/mailmanclient
>>> \git+https://gitlab.com/mailman/hyperkitty@c9fa4d4bfc295438d3e01cd93090064d004cf44d
>>> \git+https://gitlab.com/eestec/django-mailman3 \whoosh \uwsgi
>>> \psycopg2 \dj-database-url \typing \&& apk del .build-deps \&&
>>> addgroup -S -g 82 hyperkitty \&& adduser -S -u 82 -G hyperkitty
>>> hyperkitty \&& chmod u+x /usr/local/bin/docker-entrypoint.sh# Add
>>> needed files for uwsgi server + settings for django COPY
>>> assets/__init__.py /home/hyperkittyCOPY assets/manage.py
>>> /home/hyperkittyCOPY assets/urls.py /home/hyperkittyCOPY
>>> assets/wsgi.py /home/hyperkittyCOPY assets/uwsgi.ini
>>> /home/hyperkittyCOPY assets/settings.py /home/hyperkitty# Change
>>> ownership for uwsgi+django files and set execution rights for
>>> management script RUN chown -R hyperkitty /home/hyperkitty &&
>>> chmod
>>> u+x /home/hyperkitty/manage.py# Make sure we are in the correct
>>> working dir WORKDIR /home/hyperkittyEXPOSE 8080 8081# Use stop
>>> signal for uwsgi server STOPSIGNAL SIGINTENTRYPOINT
>>> ["docker-entrypoint.sh"]CMD ["uwsgi", "--ini",
>>> "/home/hyperkitty/uwsgi.ini"]
>>>
>>> On 4/27/19 7:58 PM, Abhilash Raj wrote:
>>>> On Sat, Apr 27, 2019, at 9:40 AM, Alain Kohli wrote:
>>>>> I have run "python manage.py rebuild_index" before, doesn't that do
>>>>> clear_index as well? Apart from that, I run hyperkitty in a docker
>>>>> container and didn't know fulltext_index should be persistent, so
>>>>> that
>>>>> got deleted after every version update for sure.
>>>> Which images are you using and how are you deploying them?
>>>>
>>>> You should persist fulltext_index, yes, and possibly logs if you need
>>>> them for debugging later.
>>>>
>>>> Can you paste the entire error traceback?
>>>>
>>>>> On 4/26/19 10:18 PM, Mark Sapiro wrote:
>>>>>> On 4/26/19 11:14 AM, Alain Kohli wrote:
>>>>>>> I see loads of "A string literal cannot contain NUL (0x00)
>>>>>>> characters."
>>>>>>> messages, but I haven't found missing messages in the archives
>>>>>>> yet. Not
>>>>>>> sure how that could be related, though. Apart from that I don't see
>>>>>>> anything unusual. The other jobs (quarter_hourly, hourly, etc.)
>>>>>>> seem to
>>>>>>> run and finish normally.
>>>>>> Did you upgrade from a Python 2.7 version of HyperKitty to a
>>>>>> Python 3
>>>>>> version? The Haystack/Whoosh search engine databases are not
>>>>>> compatible
>>>>>> between the two and "A string literal cannot contain NUL (0x00)
>>>>>> characters." is the symptom.
>>>>>>
>>>>>> You need to run 'python manage.py clear_index' or just remove all
>>>>>> the
>>>>>> files from the directory defined as 'PATH' under
>>>>>> HAYSTACK_CONNECTIONS in
>>>>>> your settings file (normally 'fulltext_index' in the same
>>>>>> directory that
>>>>>> contains your settings.py.
>>>>>>
>>>>> _______________________________________________
>>>>> Mailman-users mailing list -- mailman-users(a)mailman3.org
>>>>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>>>>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>>>>>
>>> _______________________________________________
>>> Mailman-users mailing list -- mailman-users(a)mailman3.org
>>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>>>
>
6 years, 4 months
Re: config incoming email in my domain cpanel
by Rafael Mora
El jue., 3 ago. 2017 a las 16:07, tlhackque via Mailman-users (<
mailman-users(a)mailman3.org>) escribió:
> You've said you're working with labbrands.com.
>
> labbrands.com has address 192.185.51.89
> mail.labbrands.com has address 192.185.51.89
> www.labbrands.com is an alias for labbrands.com.
>
> telnet proves that Exim is running on that IP address. Either you're
> running it, or hostgator is redirecting the SMTP ports.
>
> You added:
> lists.labbrands.com has address 190.145.27.66
>
> Presumably that's a different machine.
>
> So that now needs an MX record, and a firewall routing.
I added the MX record on our hostgator hosting like this:
[image: image.png]
Is it correct???
> And the SPF
> records ... and spam and virus filters, and all the other stuff you need
> to setup an independent, public-facing smtp server.
>
> But at least you won't have to relay.
>
> Good luck.
>
> On 03-Aug-17 16:55, Rafael Mora wrote:
> > El jue., 3 ago. 2017 a las 15:49, tlhackque via Mailman-users (<
> > mailman-users(a)mailman3.org>) escribió:
> >
> >> I think you're confused.
> >>
> >> You already have mail.labbrands.com set up as the MX record for
> >> labbrands.com. And it has an A record with the same address as your
> >> webserver.
> >>
> > I'm working with the hostgator mailserver, we are not running a local
> > mailserver.
> >
> > As suggested I added an A record like this:
> > [image: image.png]
> >
> > Is it correct? is it redirecting to my Ip so I can redirect it to my
> > postfix/mm3 server?
> >
> >
> >> So if you're getting e-mail on that domain, there's another e-mail
> >> server running on that IP address. You can't have 2 servers on port
> >> 25. In that case, as has been noted before, you'll need to setup a
> >> relay in that server, not a firewall redirect. Depending on your MTA,
> >> you would need to relay to your internal server. And make sure your
> >> firewall setup allows your MTA to do this.
> >>
> >> We can see it's EXIM:
> >>
> >> telnet mail.labbrands.com 25
> >> Trying 192.185.51.89...
> >> Connected to mail.labbrands.com (192.185.51.89).
> >> Escape character is '^]'.
> >> help
> >> 220-gator4137.hostgator.com ESMTP Exim 4.87 #1 Thu, 03 Aug 2017
> 15:47:48
> >> -0500
> >> 220-We do not authorize the use of this system to transport unsolicited,
> >> 220 and/or bulk e-mail.
> >> 214-Commands supported:
> >> 214 AUTH STARTTLS HELO EHLO MAIL RCPT DATA NOOP QUIT RSET HELP
> >> quit
> >> 221 gator4137.hostgator.com closing connection
> >> Connection closed by foreign host.
> >>
> >> Or, consolidate all your e-mail to one server - which is a lot easier to
> >> manage unless you have a really big operation. Postfix is probably the
> >> right choice, but requires more setup.
> >>
> >> Although Mailman3 configuration is not well documented (as you've
> >> discovered), you may want to get help from someone with more general
> >> network and mail experience. You're now into territory that is, as
> >> Simon indicated, not Mailman-specific.
> >>
> >> On 03-Aug-17 16:28, Rafael Mora wrote:
> >>> El jue., 3 ago. 2017 a las 15:26, Mark Sapiro (<mark(a)msapiro.net>)
> >> escribió:
> >>>> On 08/03/2017 01:22 PM, Rafael Mora wrote:
> >>>>> El jue., 3 ago. 2017 a las 15:18, Mark Sapiro (<mark(a)msapiro.net>)
> >>>> escribió:
> >>>>>> You need to forward port 25 for SMTP mail delivery and if you want
> the
> >>>>>> web UI (Postorius and HyperKitty) accessible from the outside, port
> 80
> >>>>>> for http and/or port 443 for https
> >>>>>> <
> >>>>>>
> >>
> https://www.iana.org/assignments/service-names-port-numbers/service-names-p…
> >>>>>>> .
> >>>>> I mean for incoming mail redirected from my hostgator hosting,
> because
> >>>> when
> >>>>> I suscribe an email address MM3 sends a confirmation email, so I have
> >> to
> >>>>> reply to be suscribed to the list.
> >>>> As I said, for mail delivery you need to forward port 25 to the
> Mailman
> >>>> server.
> >>>>
> >>> Ok so I'll redirect in my zentyal firewall the port 25 to my local
> >> centos7
> >>> with postfix and MM3 with IP 192.168.1.42. Thanks Mark.
> >>>
> >>>
> >>>> --
> >>>> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> >>>> San Francisco Bay Area, California better use your sense - B. Dylan
> >>>> _______________________________________________
> >>>> Mailman-users mailing list
> >>>> mailman-users(a)mailman3.org
> >>>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >>>>
> >> --
> >> This communication may not represent my employer's views,
> >> if any, on the matters discussed.
> >>
> >> _______________________________________________
> >> Mailman-users mailing list
> >> mailman-users(a)mailman3.org
> >> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >>
>
> --
> This communication may not represent my employer's views,
> if any, on the matters discussed.
>
> _______________________________________________
> Mailman-users mailing list
> mailman-users(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>
--
Atentamente / Best Regards
Ing. Rafael Mora
8 years, 2 months
Re: new list not working
by Christian
I am only having issues with the new list.
Below are logfile entries
Smtp.log
++++++
Mar 09 20:57:09 2023 (9401) Available AUTH mechanisms: LOGIN(builtin) PLAIN(builtin)
Mar 09 20:57:09 2023 (9401) Peer: ('127.0.0.1', 35982)
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) handling connection
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'LHLO zarathustra.ccalternatives.org'
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'MAIL FROM:<christian(a)naturalintelligence.us> SIZE=3653'
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) sender: christian(a)naturalintelligence.us
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'RCPT TO:<redlatinleg(a)lists.ccalternatives.org>'
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) recip: redlatinleg(a)lists.ccalternatives.org
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'DATA'
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) >> b'QUIT'
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) connection lost
Mar 09 20:57:09 2023 (9401) ('127.0.0.1', 35982) Connection lost during _handle_client()
Access.log
++++++++
127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/recent-threads HTTP/1.0" 200 4676 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/favorites HTTP/1.0" 200 58 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/top-threads HTTP/1.0" 200 4676 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
127.0.0.1 - - [09/Mar/2023:13:08:22 -0800] "GET /archives/list/redlatinleg(a)lists.ccalternatives.org/posted-to HTTP/1.0" 200 54 "https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
Mailman.log
++++++++
Mar 09 20:57:09 2023 (9400) ACCEPT: <032101d952c9$caf09990$60d1ccb0$(a)naturalintelligence.us>
Mar 09 13:03:42 2023 (9397) HyperKitty archived message <032101d952c9$caf09990$60d1ccb0$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…
[09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg(a)lists.ccalternatives.org HTTP/1.1" 200 497 "-" "GNU Mailman REST client v3.3.3"
[09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg(a)lists.ccalternatives.org/config HTTP/1.1" 200 3805 "-" "GNU Mailman REST client v3.3.3"
[09/Mar/2023:21:03:42 +0000] "GET /3.1/addresses/cs(a)safecomputing.org HTTP/1.1" 200 402 "-" "GNU Mailman REST client v3.3.3"
[09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg.lists.ccalternatives.org/roster/owner HTTP/1.1" 200 665 "-" "GNU Mailman REST client v3.3.3"
[09/Mar/2023:21:03:42 +0000] "GET /3.1/lists/redlatinleg.lists.ccalternatives.org/roster/moderator HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.3"
[09/Mar/2023:21:03:42 +0000] "GET /3.1/users/christian(a)naturalintelligence.us HTTP/1.1" 200 297 "-" "GNU Mailman REST client v3.3.3"
[09/Mar/2023:21:08:22 +0000] "GET /3.1/users/58f1e71ff425473d935fda2a7f9c2171 HTTP/1.1" 200 395 "-" "GNU Mailman REST client v3.3.3"
mailmansuite.log
++++++++++++
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django_mailman3/lib/paginator.py", line 56, in count
return self.function(count=0, page=1).total_size
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/postorius/views/list.py", line 962, in _get_list_page
advertised=advertised, mail_host=mail_host, count=count, page=page)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/client.py", line 210, in get_list_page
return Page(self._connection, url, MailingList, count, page)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/page.py", line 37, in __init__
self._create_page()
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/page.py", line 62, in _create_page
response, content = self._connection.call(self._build_url())
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/connection.py", line 169, in call
'Could not connect to Mailman API: ', repr(e))
mailmanclient.restbase.connection.MailmanConnectionError: ('Could not connect to Mailman API: ', 'ConnectionError(MaxRetryError("HTTPConnectionPool(host=\'localhost\', port=8001): Max retries exceeded with url: /3.1/lists?advertised=true&count=0&page=1 (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7f910b1d7da0>: Failed to establish a new connection: [Errno 111] Connection refused\'))"))')
ERROR 2023-03-08 13:52:33,078 623 django.request Service Unavailable: /mailman3/lists/
ERROR 2023-03-08 13:52:33,078 623 django.request Service Unavailable: /mailman3/lists/
INFO 2023-03-08 14:12:32,804 5421 hyperkitty.views.mailman Archived message <01e001d9520a$3ca64320$b5f2c960$(a)sewagesludgeactionnetwork.com> to https://lists.ccalternatives.org/archives/list/ssan@lists.ccalternatives.or…
INFO 2023-03-08 14:16:54,400 5423 hyperkitty.views.mailman Archived message <01d801d95207$e34e52b0$a9eaf810$(a)sewagesludgeactionnetwork.com> to https://lists.ccalternatives.org/archives/list/ssan@lists.ccalternatives.or…
INFO 2023-03-08 15:01:16,764 8917 hyperkitty.views.mailman Archived message <021201d95210$d2818c00$7784a400$(a)safecomputing.org> to https://lists.ccalternatives.org/archives/list/medialist@lists.ccalternativ…
ERROR 2023-03-08 17:18:00,624 8915 django.security.DisallowedHost Invalid HTTP_HOST header: 'www.vsooth.com'. You may need to add 'www.vsooth.com' to ALLOWED_HOSTS.
ERROR 2023-03-09 06:45:21,849 8916 django.security.DisallowedHost Invalid HTTP_HOST header: 'icook.tw'. You may need to add 'icook.tw' to ALLOWED_HOSTS.
INFO 2023-03-09 07:46:33,940 8915 hyperkitty.lib.mailman Imported the new list redlatinleg(a)lists.ccalternatives.org from Mailman
INFO 2023-03-09 08:01:35,590 8916 hyperkitty.views.mailman Archived message <02ab01d9529f$967afae0$c370f0a0$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…
INFO 2023-03-09 08:27:34,189 8919 hyperkitty.views.mailman Archived message <02c301d952a3$37279c70$a576d550$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…
INFO 2023-03-09 08:50:56,941 8916 hyperkitty.views.mailman Archived message <02cc01d952a6$7bdfbed0$739f3c70$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…
INFO 2023-03-09 13:03:42,385 8916 hyperkitty.views.mailman Archived message <032101d952c9$caf09990$60d1ccb0$(a)naturalintelligence.us> to https://lists.ccalternatives.org/archives/list/redlatinleg@lists.ccalternat…
From: Zaheer Abbas <zaheer.psg(a)gmail.com>
Sent: Thursday, March 9, 2023 9:00 AM
To: csa(a)web-analysts.net; Christian <eric(a)safecomputing.org>
Cc: mailman-users(a)mailman3.org
Subject: Re: [MM3-users] new list not working
Are you having issues with only this list or all lists?
Did you configure and test smtp configuration?
Check for mailman and smtp log files, usually placed in /var/log/mailman3/
On Thu, Mar 9, 2023 at 4:51 PM Christian via Mailman-users <mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org> > wrote:
I created a new list using via the form. I sent a test message to the new
list and it appears in the list archive. However I am not receiving the
distribution.
What should I check for please?
Thank you ahead of time!
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>
To unsubscribe send an email to mailman-users-leave(a)mailman3.org <mailto:mailman-users-leave@mailman3.org>
https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
This message sent to zaheer.psg(a)gmail.com <mailto:zaheer.psg@gmail.com>
--
Zaheer Abbas
Cell:+44-740-5511766
Personal Web: www.zaheerabbas.info <http://www.zaheerabbas.info>
'SAVE WATER ~ SAVE ENERGY ~ ~ SAVE EARTH
2 years, 7 months
Re: REST API notes
by Barry Warsaw
On Jun 28, 2017, at 01:47 AM, tlhackque--- via Mailman-users wrote:
>As a beginner with the REST API, I found it necessary to read code and
>experiment to figure out how to get things done.
Thanks for the feedback. Other than the work supporting Postorius and
HyperKitty, I think you've done some of the more extensive exploration of the
REST API, so your comments here (and greatly appreciated bug reports) are
quite valuable, and I think will help us make the REST API both more usable
and more discoverable.
It's important to point out some history of the REST API. Originally, we used
the restish library (which was Python 2 and eventually abandoned upstream).
That choice informed the essentially dynamic nature of the REST API. Which
means that unlike many other systems, we don't have -nor do I think we could
if we wanted to- easily and automatically generate a static description of the
REST API.
When I switched to Falcon (which is both Python 3 compatible and well
maintained upstream), I had to reimplement what I called "object based
traversal". Falcon by default supports a routes based traversal mechanism,
similar to Flask and other systems. But by plugging in (originally, by
monkeypatching, but now through an official Falcon API) support for object
based traversal, I was able to port the extensive existing REST implementation
to the new underlying library fairly easily.
All this to say that some of the problems you describe are due to historical
accident. And that's served us well(-ish) while we only had to support
mailmanclient and P/HK. But as the REST API is used in more places, those
deficiencies are becoming more glaring. I think if we ever make progress on
Lemme (our authenticating REST proxy), we'll likely encounter the same
problems. But at this point, it's infeasible to completely reimplement the
REST machinery.
>There are a number of implementation choices that are unusual.
No doubt. :)
>There is some documentation in src/mailman/rest/docs. It's oriented toward
>Python and intermixes the mailman client & REST API, apparently trying to
>show equivalence.
Much of that is because these are doctests, i.e. testable documentation. It's
a tradeoff between making them useful as docs but also testable without too
much clutter. It's very much oriented toward Python because that makes
testability easier. It would be good to have documentation for pure HTTP/JSON
consumers, but it would be imperative for that also to be testable so we're
sure it remains accurate, aside from any requirements to keep the docs in
sync. Suggestions and contributions would be very much welcome here.
>The REST API has a hybrid interface: Requests are made with
>application/x-www-form-urlencoded POST, PUT, PATCH and delete http requests.
>Requests are also accepted as parameter strings.
>
>The responses are JSON. (This is rather surprising - one would expect JSON
>requests - and I hope someday they'll be accepted, as the split complicates
>clients. I suspect this evolved to simplify the initial Web GUI client
>(Postorious), but it precludes using standard JSON-in/JSON-out client
>libraries.)
Absolutely. I want to support JSON-in/JSON-out. Again, the current mismatch
is due to historical implementation decisions, but there's nothing in
principle preventing us from accepting JSON in, afaict. We'd have to continue
to accept both, for backward compatibility reasons.
>Error responses are often misidentified as content-type: application/json,
>but actually contain a text/plain error message. This isn't universally
>true; for example the 401 response actually IS JSON. A client has to guess
>and handle decoding exceptions.
That's clearly a bug. The Content-Type should always be accurate. It would
be a new feature to also support JSON error responses.
>The API presents resources hierarchically, rooted at '/'. The top level
>resources include /users, /lists. /addresses, /domains, /system
>
>The next level is a resource id.
Kind of. It all depends on what the top-level resource is returned. This
makes sense when you think about the object based traversal machinery.
Each path component points internally to an object, and the subsequent path
component is handled by that object. If that returns another object, then the
next remaining path component is consumed by *that* object. And so on until
we reach the end of the path. Then the object at the leaf responds to the
HTTP command, and each object knows how to represent itself as JSON, and it
knows its canonical location within the resource tree (there can be multiple
paths to any particular object, but there's always one canonical location).
>There is a an id which is stable for the lifetime of an object, and a current
>name (which can be changed). For example, a list has a name like
>mylist(a)example.com, and a list_id of mylist.example.com. But if you change
>the list name, it becomes mynewname(a)example.net, while the list_id remains
>mylist.example.com.
Lists are weird in that they have two identifiers. One is the posting
address, what we internally (and slightly incorrectly) call the "fully
qualified list name". I don't particularly like that nomenclature anymore,
but we live with it.
A posting address can change, e.g. if you rename or rehome a list.
The (RFC 2919) List-ID is assigned when the mailing list is created and it's
immutable. Section 4 discourages changing the List-ID and Mailman takes that
as a requirement. Rename a list or rehome it to a different domain on the
same server and the List-ID will never change.
Again for historical reasons, many APIs both internal and external used the
posting address to identify the list, but this is wrong exactly because that
can change. I've slowly been converting APIs to accept both the posting
address and the List-ID when identifying the mailing list. New APIs generally
accept only the List-ID. Bottom line, it's best to use the List-ID.
>Lists are associated with "domains", which are the "domain part" of the
>list's address. That is, the part after the @. This is sometimes referred
>to as the "mail host", but there need not be a real host.
Yep, again historical nomenclature.
>A domain has to be created before you can create a list with an address in
>that domain.
Some APIs (e.g. `mailman create` CLI) will create the domain automatically if
it doesn't already exist, and unless you disable that explicitly.
>Resources are created with a POST to their top-level resource. To create a
>domain, post to /domains with mail_host => the domain, and (optionally)
>description => a description for the GUI. The response isn't JSON as one
>would expect. In fact, it's an empty application/json response with a 201
>status.
>
>To create a list, one POSTs to /lists. This post takes a restricted set of
>parameters; in the case of a list, just its fqdn_listname, (and an optional
>style - which isn't well defined). The response isn't JSON as one would
>expect. The Location header of the response contains a URL of the new list.
What else would you expect? From my reading of books like RESTful Web
Services (admittedly a long while ago), that's exactly the proper response to
an appending POST. Return a 201 with a Location header to the new resource
and empty content.
>To configure the list, you have to follow up with a PUT or PATCH to that URL
>+ /config This is where you can set the description, posting policy, etc.
>It's unrealistic to do a PUT; even if you're cloning another list, there's no
>programatic way to determine which attributes are writable. PATCH what you
>know...
Mailing list resources are somewhat unique in that they probably have the most
properties of any resource/object in the system. That's not surprising if you
think about it, but it does make PUTing to a mailing list more or less
impractical. That's certainly not true of other, smaller resources though,
and of course, you still want symmetry there (plus, implementation-wise, it's
almost a no brainer to support both PUT and PATCH).
>Members are associated with e-mail addresses - which belong to Users.
Users can have multiple addresses, addresses can be linked to only one user,
but may be unlinked, and members associate "subscribers" to mailing lists,
where a subscriber is either a user-with-preferred-address, or an address.
>You create a user by posting to /users with email => the email address, and
>optionally display_name => the name string. (A user can also be implicitly
>created by subscribing an e-mail address, but that gets confusing.) The
>e-mail address is the primary email address for the user. More later.
>Again, you get a Location header in the response, which you can use to PATCH
>/preferences to set delivery_status, etc. .These preferences are part of a
>hierarchy - many have a system default, a list default, the user default, and
>a subscription to that list value. You can find the User associated with a
>list by a GET of /addresses/address(a)example.net.
Technically, the returned user isn't necessarily associated with a list. It
*may* be subscribed to the list, but that relationship is represented by a
member.
>This GET returns two URLs: user => the user owning this address, and
>self_link => the address object.
*All* resources in the REST API have a self_link, and while that may seem
redundant, it's not. As mentioned above, you can take various paths through
the resource tree to get to a particular resource, but regardless of that,
every resource has exactly one canonical location. That location is
represented by the self_link.
>Note also that there are both an email and original_email attribute. The
>latter preserves case. The former is used internally by Mailman as a
>resource key, etc. (Though exactly what happens if John(a)example.net and
>john(a)example.net both subscribe is unclear.)
They can't. Mailman is case-preserving case-insensitive for email addresses.
Technically speaking, john(a)example.com and JOHN(a)example.com can be different
mailboxes, but that never happens in practice anymore, and Mailman has always
explicitly treated them as the same address. This goes back to the earliest
days of Mailman.
>Once a list is configured, you can add members. This requires the list_id -
>which you don't (officially) have. So you do a GET on the list resource, to
>get the list_id. Then you can subscribe a member with list_id => (the list
>id), subscriber => email address. Optionally you can
>pre_verify/confirm/approve the member and/or add a display_name.. POST to
>/members. Again, you get a Location header back. You can't specify
>everything you might like. as a creation attribute. So you may have to PATCH
>the member to, for example, set the moderation_status.
This is true, but it also kind of makes sense if you grok the way preferences
work, i.e. hierarchically as you describe above. TBH, I don't particularly
like the way preferences are modeled either internally or through the REST
API, but I haven't been able to come up with anything better.
>You may also need to PATCH the member /preferences if you want to set
>list-level delivery status, etc.
All true. In each of those cases, you may be talking to a different
preference resource.
>Consider adding at least an owner when creating a list.
I think that would be a useful addition to the REST API for list creation.
>One challenge is that almost everything requires multiple REST operations to
>set up. But REST (by definition) is stateless. So the best you can do is
>order operations & hope.
I don't understand this. REST is stateless, but not all HTTP operations are
idempotent. POST certainly isn't, by definition, so if you use it to create a
new resource under a collection, you clearly cannot modify that resource
through PUT or PATCH until the resource is created, which only happens when
POST succeeds.
>The mailman client examples refer to transactions (e.g in users.rst, there is
>'transaction.commit(); - but REST can't hold state. It does appear that the
>server uses DB transactions to ensure that any given REST operation is ACID,
>but the composite operations (e.g. create a list and set it's config) can not
>be Atomic. This is an architectural flaw in the API.
I think what you're getting at is that some POSTs to create new resources do
not accept the same parameters as the subsequent PUT or PATCH on the newly
created resource. I think that may be true, but it's not universally so.
E.g. POST on /domain creates a new domain, returning the location of the new
resource. You can provide both a description and an owner, and you can also
provide a mail_host. But mail_host is immutable, and PATCH or PUT support
changing all mutable attributes on the domain.
That should be the general rule; if an attribute on a resource is mutable, you
should be able to PATCH and PUT it, *and* you should be able to specify all
mutable attributes, plus some immutable ones on the POST that creates the
resource. GET should return all attributes, mutable or immutable. This rule
may not be strictly adhered to for all resources and collections, but I would
consider that a bug, not an architectural flaw.
AFAIK, this is all working as any well-defined web service should work.
>Symbolic names (which are required) for attributes are in the
>src/mailman/interfaces/(class).py.
Please note that the interfaces under this package are for the *internal* API,
which often, but not always, is exposed in the REST API. These two APIs serve
different purposes so they cannot be a one-to-one mapping, and there are many
resources in the REST API that don't correspond directly to internal objects,
and there are many internal objects that are not exposed to the REST API.
>What you can get/set comes from src/mailman/rest/(class).py.
Yes, so clients of the REST API are served by objects in this package.
Plugins and other internal operations are served by the objects that implement
interfaces in the src/mailman/interfaces package. This is a strict separation
of concerns, and it mirrors other 'external facing' interfaces, of which the
command line is another example.
Hope that helps.
-Barry
8 years, 3 months
Re: Migrating mailman3 to latest ubuntu lts
by Helio Loureiro
And using the module manually works...
(venv) mailman@new-server ~ (v3.3.9) [0|127]> *ipython3*
/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py:949:
UserWarning: Attempting to work in a virtualenv. If you encounter problems,
please install IPython inside the virtualenv.
warn(
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: *import MySQLdb*
(venv) mailman@new-sever ~ (v3.3.9)> *fg*
Send job 1, 'ipython3' to foreground
In [3]: *con = MySQLdb.connect("localhost", "mailman3web",
"*****************", "mailman3web")*
In [4]:
*cursor = con.cursor()*
In [5]: *cursor.execute("show tables")*
Out[5]: 34
In [6]: *print(cursor.fetchall())*
(('account_emailaddress',), ('account_emailconfirmation',),
('auth_group',), ('auth_group_permissions',), ('auth_permission',),
('auth_user',), ('auth_user_groups',), ('auth_user_user_permissions',),
('django_admin_log',), ('django_content_type',),
('django_mailman3_maildomain',), ('django_mailman3_profile',),
('django_migrations',), ('django_q_ormq',), ('django_q_schedule',),
('django_q_task',), ('django_session',), ('django_site',),
('hyperkitty_attachment',), ('hyperkitty_email',),
('hyperkitty_favorite',), ('hyperkitty_lastview',),
('hyperkitty_mailinglist',), ('hyperkitty_profile',),
('hyperkitty_sender',), ('hyperkitty_tag',), ('hyperkitty_tagging',),
('hyperkitty_thread',), ('hyperkitty_threadcategory',),
('hyperkitty_vote',), ('socialaccount_socialaccount',),
('socialaccount_socialapp',), ('socialaccount_socialapp_sites',),
('socialaccount_socialtoken',))
In [7]: *cursor.execute("select * from account_emailaddress")*
Out[7]: 2530
In [8]: *print(cursor.fetchall())*
So what can I do? Is there a way to troubleshoot the root cause of this
issue?
Best Regards,
Helio Loureiro
https://helio.loureiro.eng.br
https://github.com/helioloureiro
https://mastodon.social/@helioloureiro
On Tue, 19 Dec 2023 at 11:11, Helio Loureiro <helio(a)loureiro.eng.br> wrote:
> Hi,
>
> No luck :(
>
> (venv) mailman@new-server ~ (v3.3.9)> *pip freeze | egrep
> "mailman-web|django-mailman3|django-allauth"*
> django-allauth==0.59.0
> django-mailman3==1.3.11
> mailman-web==0.0.8
> (venv) mailman@new-server ~ (v3.3.9)> *pip install -U
> django-allauth==0.58.0*
> Collecting django-allauth==0.58.0
> Downloading django-allauth-0.58.0.tar.gz (861 kB)
> ---------------------------------------- 861.7/861.7 KB 9.4 MB/s eta
> 0:00:00
> Installing build dependencies ... done
> Getting requirements to build wheel ... done
> Installing backend dependencies ... done
> Preparing metadata (pyproject.toml) ... done
> Requirement already satisfied: requests-oauthlib>=0.3.0 in
> ./venv/lib/python3.10/site-packages (from django-allauth==0.58.0) (1.3.1)
> Requirement already satisfied: Django>=3.2 in
> ./venv/lib/python3.10/site-packages (from django-allauth==0.58.0) (4.1.13)
> Requirement already satisfied: pyjwt[crypto]>=1.7 in
> ./venv/lib/python3.10/site-packages (from django-allauth==0.58.0) (2.8.0)
> Requirement already satisfied: requests>=2.0.0 in
> ./venv/lib/python3.10/site-packages (from django-allauth==0.58.0) (2.31.0)
> Requirement already satisfied: python3-openid>=3.0.8 in
> ./venv/lib/python3.10/site-packages (from django-allauth==0.58.0) (3.2.0)
> Requirement already satisfied: asgiref<4,>=3.5.2 in
> ./venv/lib/python3.10/site-packages (from
> Django>=3.2->django-allauth==0.58.0) (3.7.2)
> Requirement already satisfied: sqlparse>=0.2.2 in
> ./venv/lib/python3.10/site-packages (from
> Django>=3.2->django-allauth==0.58.0) (0.4.4)
> Requirement already satisfied: cryptography>=3.4.0 in
> ./venv/lib/python3.10/site-packages (from
> pyjwt[crypto]>=1.7->django-allauth==0.58.0) (41.0.7)
> Requirement already satisfied: defusedxml in
> ./venv/lib/python3.10/site-packages (from
> python3-openid>=3.0.8->django-allauth==0.58.0) (0.7.1)
> Requirement already satisfied: urllib3<3,>=1.21.1 in
> ./venv/lib/python3.10/site-packages (from
> requests>=2.0.0->django-allauth==0.58.0) (2.1.0)
> Requirement already satisfied: charset-normalizer<4,>=2 in
> ./venv/lib/python3.10/site-packages (from
> requests>=2.0.0->django-allauth==0.58.0) (3.3.2)
> Requirement already satisfied: idna<4,>=2.5 in
> ./venv/lib/python3.10/site-packages (from
> requests>=2.0.0->django-allauth==0.58.0) (3.6)
> Requirement already satisfied: certifi>=2017.4.17 in
> ./venv/lib/python3.10/site-packages (from
> requests>=2.0.0->django-allauth==0.58.0) (2023.11.17)
> Requirement already satisfied: oauthlib>=3.0.0 in
> ./venv/lib/python3.10/site-packages (from
> requests-oauthlib>=0.3.0->django-allauth==0.58.0) (3.2.2)
> Requirement already satisfied: typing-extensions>=4 in
> ./venv/lib/python3.10/site-packages (from
> asgiref<4,>=3.5.2->Django>=3.2->django-allauth==0.58.0) (4.9.0)
> Requirement already satisfied: cffi>=1.12 in
> ./venv/lib/python3.10/site-packages (from
> cryptography>=3.4.0->pyjwt[crypto]>=1.7->django-allauth==0.58.0) (1.16.0)
> Requirement already satisfied: pycparser in
> ./venv/lib/python3.10/site-packages (from
> cffi>=1.12->cryptography>=3.4.0->pyjwt[crypto]>=1.7->django-allauth==0.58.0)
> (2.21)
> Building wheels for collected packages: django-allauth
> Building wheel for django-allauth (pyproject.toml) ... done
> Created wheel for django-allauth:
> filename=django_allauth-0.58.0-py3-none-any.whl size=1157319
> sha256=a430c552101d1ad47bc00b16d1c1d6df728afacdd13823927b4cbfb02c35dbfc
> Stored in directory:
> /local/mailman/.cache-ubuntu-22.04/pip/wheels/55/0a/79/e199827a18f310906c2a90b0e92b89c41daf21d2a502db6710
> Successfully built django-allauth
> Installing collected packages: django-allauth
> Attempting uninstall: django-allauth
> Found existing installation: django-allauth 0.59.0
> Uninstalling django-allauth-0.59.0:
> Successfully uninstalled django-allauth-0.59.0
> Successfully installed django-allauth-0.58.0
> (venv) mailman@new-server ~ (v3.3.9)> *mailman-web migrate*
> System check identified some issues:
>
> WARNINGS:
> account.EmailAddress: (models.W036) MariaDB does not support unique
> constraints with conditions.
> HINT: A constraint won't be created. Silence this warning if you don't
> care about it.
> account.EmailAddress: (models.W043) MariaDB does not support indexes on
> expressions.
> HINT: An index won't be created. Silence this warning if you don't care
> about it.
> Operations to perform:
> Apply all migrations: account, admin, auth, contenttypes,
> django_mailman3, django_q, hyperkitty, postorius, sessions, sites,
> socialaccount
> Running migrations:
> Applying account.0004_alter_emailaddress_drop_unique_email...Traceback
> (most recent call last):
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> line 89, in _execute
> return self.cursor.execute(sql, params)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/mysql/base.py",
> line 75, in execute
> return self.cursor.execute(query, args)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line
> 179, in execute
> res = self._query(mogrified_query)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line
> 330, in _query
> db.query(q)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/connections.py",
> line 257, in query
> _mysql.connection.query(self, query)
> MySQLdb.OperationalError: (2013, 'Lost connection to MySQL server during
> query')
>
> The above exception was the direct cause of the following exception:
>
> Traceback (most recent call last):
> File "/local/mailman/venv/bin/mailman-web", line 8, in <module>
> sys.exit(main())
> File
> "/local/mailman/venv/lib/python3.10/site-packages/mailman_web/manage.py",
> line 90, in main
> execute_from_command_line(sys.argv)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/__init__.py",
> line 446, in execute_from_command_line
> utility.execute()
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/__init__.py",
> line 440, in execute
> self.fetch_command(subcommand).run_from_argv(self.argv)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/base.py",
> line 402, in run_from_argv
> self.execute(*args, **cmd_options)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/base.py",
> line 448, in execute
> output = self.handle(*args, **options)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/base.py",
> line 96, in wrapped
> res = handle_func(*args, **kwargs)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py",
> line 349, in handle
> post_migrate_state = executor.migrate(
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/executor.py",
> line 135, in migrate
> state = self._migrate_all_forwards(
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/executor.py",
> line 167, in _migrate_all_forwards
> state = self.apply_migration(
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/executor.py",
> line 252, in apply_migration
> state = migration.apply(state, schema_editor)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/migration.py",
> line 130, in apply
> operation.database_forwards(
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/migrations/operations/fields.py",
> line 235, in database_forwards
> schema_editor.alter_field(from_model, from_field, to_field)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py",
> line 788, in alter_field
> self._alter_field(
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py",
> line 858, in _alter_field
> self.execute(self._delete_unique_sql(model, constraint_name))
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/base/schema.py",
> line 199, in execute
> cursor.execute(sql, params)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> line 67, in execute
> return self._execute_with_wrappers(
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> line 80, in _execute_with_wrappers
> return executor(sql, params, many, context)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> line 84, in _execute
> with self.db.wrap_database_errors:
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/utils.py", line
> 91, in __exit__
> raise dj_exc_value.with_traceback(traceback) from exc_value
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> line 89, in _execute
> return self.cursor.execute(sql, params)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/django/db/backends/mysql/base.py",
> line 75, in execute
> return self.cursor.execute(query, args)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line
> 179, in execute
> res = self._query(mogrified_query)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/cursors.py", line
> 330, in _query
> db.query(q)
> File
> "/local/mailman/venv/lib/python3.10/site-packages/MySQLdb/connections.py",
> line 257, in query
> _mysql.connection.query(self, query)
> django.db.utils.OperationalError: (2013, 'Lost connection to MySQL server
> during query')
> (venv) mailman@new-server ~ (v3.3.9) [0|1]> *more
> /etc/mailman3/settings.py*
> # Mailman Web configuration file.
> # /etc/mailman3/settings.py
>
> # Get the default settings.
> from mailman_web.settings.base import *
> from mailman_web.settings.mailman import *
>
> # Settings below supplement or override the defaults.
>
> #: Default list of admins who receive the emails from error logging.
> ADMINS = (
> ('Mailman Suite Admin', 'root@localhost'),
> )
>
> # Postgresql database setup.
> DATABASES = {
> 'default': {
> 'ENGINE': 'django.db.backends.mysql',
> 'NAME': 'mailman3web',
> 'USER': 'mailman3web',
> # TODO: Replace this with the password.
> 'PASSWORD': '***********',
> 'HOST': 'localhost',
> # PORT: set to empty string for default.
> 'PORT': '3306',
> # OPTIONS: Extra parameters to use when connecting to the database.
> #'OPTIONS': {
> # Set sql_mode to 'STRICT_TRANS_TABLES' for MySQL. See
> # https://docs.djangoproject.com/en/1.11/ref/
> # databases/#setting-sql-mode
> # 'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
> # 'charset': 'utf8mb4',
> #},
>
> }
> }
>
> # 'collectstatic' command will copy all the static files here.
> # Alias this location from your webserver to `/static`
> STATIC_ROOT = '/local/mailman/web/static'
>
> # enable the 'compress' command.
> COMPRESS_ENABLED = True
>
> # Make sure that this directory is created or Django will fail on start.
> LOGGING['handlers']['file']['filename'] =
> '/local/mailman/web/logs/mailmanweb.log'
>
> #: See https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
> ALLOWED_HOSTS = [
> "localhost", # Archiving API from Mailman, keep it.
> "127.0.0.1",
> # "lists.your-domain.org",
> # Add here all production domains you have.
> "*"
> ]
>
> #: See
> https://docs.djangoproject.com/en/dev/ref/settings/#csrf-trusted-origins
> (venv) mailman@new-server ~ (v3.3.9)> *mysql -umailman3web -p -h
> localhost mailman3web*
> Enter password:
> Reading table information for completion of table and column names
> You can turn off this feature to get a quicker startup with -A
>
> Welcome to the MariaDB monitor. Commands end with ; or \g.
> Your MariaDB connection id is 32
> Server version: 10.6.12-MariaDB-0ubuntu0.22.04.1 Ubuntu 22.04
>
> Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
>
> Type 'help;' or '\h' for help. Type '\c' to clear the current input
> statement.
>
> MariaDB [mailman3web]> show tables;
> +-------------------------------+
> | Tables_in_mailman3web |
> +-------------------------------+
> | account_emailaddress |
> | account_emailconfirmation |
> | auth_group |
> | auth_group_permissions |
> | auth_permission |
> | auth_user |
> | auth_user_groups |
> | auth_user_user_permissions |
> | django_admin_log |
> | django_content_type |
> | django_mailman3_maildomain |
> | django_mailman3_profile |
> | django_migrations |
> | django_q_ormq |
> | django_q_schedule |
> | django_q_task |
> | django_session |
> | django_site |
> | hyperkitty_attachment |
> | hyperkitty_email |
> | hyperkitty_favorite |
> | hyperkitty_lastview |
> | hyperkitty_mailinglist |
> | hyperkitty_profile |
> | hyperkitty_sender |
> | hyperkitty_tag |
> | hyperkitty_tagging |
> | hyperkitty_thread |
> | hyperkitty_threadcategory |
> | hyperkitty_vote |
> | socialaccount_socialaccount |
> | socialaccount_socialapp |
> | socialaccount_socialapp_sites |
> | socialaccount_socialtoken |
> +-------------------------------+
> 34 rows in set (0.000 sec)
>
> Best Regards,
> Helio Loureiro
> https://helio.loureiro.eng.br
> https://github.com/helioloureiro
> https://mastodon.social/@helioloureiro
>
>
> On Mon, 18 Dec 2023 at 17:11, Mark Sapiro <mark(a)msapiro.net> wrote:
>
>> On 12/18/23 6:24 AM, Helio Loureiro wrote:
>> > Hi,
>> >
>> > Indeed it was the configuration. It was placed into
>> > /etc/mailman3/mailman-web.py. After a I changed to
>> > /etc/mailman3/settings.py a few things advanced a little bit more.
>> >
>> > I had to figure out how to fix mysqlclient installation since there
>> isn't a
>> > mention about it and the simple "pip install mysqclient" was breaking
>> with
>> > pkg-config issues. But it did work at the end.
>> >
>> > Now I can see further messages on mailman3-web than before.
>> >
>> > (venv) mailman@new-server ~ (v3.3.9)> mailman-web migrate
>> > System check identified some issues:
>> >
>> > WARNINGS:
>> > account.EmailAddress: (models.W036) MariaDB does not support unique
>> > constraints with conditions.
>> > HINT: A constraint won't be created. Silence this warning if you don't
>> care
>> > about it.
>> > account.EmailAddress: (models.W043) MariaDB does not support indexes on
>> > expressions.
>> > HINT: An index won't be created. Silence this warning if you don't care
>> > about it.
>> > Operations to perform:
>> > Apply all migrations: account, admin, auth, contenttypes,
>> > django_mailman3, django_q, hyperkitty, postorius, sessions, sites,
>> > socialaccount
>> > Running migrations:
>> > Applying
>> account.0004_alter_emailaddress_drop_unique_email...Traceback
>> > (most recent call last):
>>
>>
>> I'm not sure why there would be an issue with this migration, but there
>> is a possible compatibility issue depending on how you installed things.
>>
>> django-mailman3<=1.3.11 is not compatible with django-allauth>=0.58.
>>
>> In your venv, try
>> ```
>> pip install django-allauth\<0.58
>> ```
>>
>> --
>> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
>> San Francisco Bay Area, California better use your sense - B. Dylan
>>
>> _______________________________________________
>> Mailman-users mailing list -- mailman-users(a)mailman3.org
>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>> Archived at:
>> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>>
>> This message sent to helio(a)loureiro.eng.br
>>
>
1 year, 9 months
Re: using SSH/TLS with external MTA
by Roland Giesler
On 2024/07/28 15:26, Roland Giesler via Mailman-users wrote:
> On 2024/07/28 00:39, Mark Sapiro wrote:
>> On 7/27/24 15:16, Roland Giesler via Mailman-users wrote:
>>>
>>> The server runs power-mailinabox, which uses Postfix indeed. All my
>>> mail, and quite a few others' mail is sent from the using SMTP-Auth
>>> on port 465. Actually, this message is sent via that server too.
>>> So it's clear to me that MM3 is not authenticating for some reason.
>>
>> What do you see in Mailman's smtp.log and in the mail.log on the mail
>> server?
>
> The smtp log is in syslog...
I also see this in /var/log/mailman3/web/mailman-web.log
*** Starting uWSGI 2.0.20-debian (64bit) on [Sun Jul 28 15:27:48 2024] ***
compiled with version: 11.2.0 on 21 March 2022 11:00:44
os: Linux-5.15.108-1-pve #1 SMP PVE 5.15.108-2 (2023-07-20T10:06Z)
nodename: mailman
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 1
current working directory: /
detected binary path: /usr/bin/uwsgi-core
setgid() to 33
setuid() to 33
chdir() to /usr/share/mailman3-web
your processes number limit is 513922
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /run/mailman3-web/uwsgi.sock fd 4
Python version: 3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0]
Python main interpreter initialized at 0x561ce813d1d0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166752 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
/usr/lib/python3/dist-packages/django_q/conf.py:139: UserWarning: Retry
and timeout are misconfigured. Set retry larger than timeout,
failure to do so will cause the tasks to be retriggered before
completion.
See
https://django-q.readthedocs.io/en/latest/configure.html#retry for details.
warn(
WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter
0x561ce813d1d0 pid: 113 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 113)
spawned uWSGI worker 1 (pid: 339, cores: 2)
[uwsgi-daemons] spawning "python3 manage.py qcluster" (uid: 33 gid: 33)
/usr/lib/python3/dist-packages/django_q/conf.py:139: UserWarning: Retry
and timeout are misconfigured. Set retry larger than timeout,
failure to do so will cause the tasks to be retriggered before
completion.
See
https://django-q.readthedocs.io/en/latest/configure.html#retry for details.
warn(
System check identified some issues:
WARNINGS:
django_mailman3.MailDomain: (models.W042) Auto-created primary key used
when not defining a primary key type, by default
'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
DjangoMailman3Config.default_auto_field attribute to point to a subclass
of AutoField, e.g. 'django.db.models.BigAutoField'.
django_mailman3.Profile: (models.W042) Auto-created primary key used
when not defining a primary key type, by default
'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
DjangoMailman3Config.default_auto_field attribute to point to a subclass
of AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Attachment: (models.W042) Auto-created primary key used when
not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Email: (models.W042) Auto-created primary key used when not
defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Favorite: (models.W042) Auto-created primary key used when
not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.LastView: (models.W042) Auto-created primary key used when
not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.MailingList: (models.W042) Auto-created primary key used when
not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Profile: (models.W042) Auto-created primary key used when not
defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Tag: (models.W042) Auto-created primary key used when not
defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Tagging: (models.W042) Auto-created primary key used when not
defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Thread: (models.W042) Auto-created primary key used when not
defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.ThreadCategory: (models.W042) Auto-created primary key used
when not defining a primary key type, by default
'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
hyperkitty.Vote: (models.W042) Auto-created primary key used when not
defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
HyperKittyConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
postorius.EmailTemplate: (models.W042) Auto-created primary key used
when not defining a primary key type, by default
'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the
PostoriusConfig.default_auto_field attribute to point to a subclass of
AutoField, e.g. 'django.db.models.BigAutoField'.
15:27:55 [Q] INFO Q Cluster low-vermont-harry-fillet starting.
15:27:55 [Q] INFO Process-1 guarding cluster low-vermont-harry-fillet
15:27:55 [Q] INFO Q Cluster low-vermont-harry-fillet running.
15:27:55 [Q] INFO Process-1:3 pushing tasks at 386
15:27:55 [Q] INFO Process-1:2 monitoring at 385
15:27:55 [Q] INFO Process-1:1 ready for work at 384
1 year, 2 months
Mailman3 in dockers on EC2 - running out of memory and puking ev.ry.where.
by roger.hislop＠is.co.za
Hi all -- really need some help. I'm tech literate, but definitely not a sysadmin or dev.
I'm following the tutorial at "Olay's Farmland" at https://www.olay.xyz/2018/01/01/deploy-mailman-3-on-aws-using-docker/ which seems to be a clone of the same content here https://xiaoxing.us/2018/01/01/deploy-mailman-3-on-aws-using-docker/
Easy enough to get most of the way ... set up EC2 instance, configure SES and add DNS settings with DKIM etc. Set up Postfix, test it. All working groovy.
I run through the how to all the way through setting up the containers and firing it up. This is where things start to get funky.
Once the docker-compose is run, I run "curl http://172.19.199.3:8000/postorius/lists/" to test. First time it said it couldn't connect to database. So I nuked and restarted from scratch. Second time it said the same thing... was reading around the place and tried again, and then reran that curl and it worked -- gave me html from a page. But the VM started running slowly --taking forever to follow keypresses.
Eventually became unresponsive. So I rebooted the machine via AWS console, and docker ps showed only the postgres db running. docker-compose ps shows
Name Command State Ports
------------------------------------------------------------------------------------------
docker-mailman_database_1 docker-entrypoint.sh postgres Up 5432/tcp
mailman-core docker-entrypoint.sh maste ... Exit 255 8001/tcp, 8024/tcp
mailman-web docker-entrypoint.sh uwsgi ... Exit 255 8000/tcp, 8080/tcp
So rebuilt the container -- same thing. It looks and feels like its running out of memory.
I used the compose file from https://gist.githubusercontent.com/Yexiaoxing/833bfcc5d3e4e0c06a8b7f0bac7c4… (with appropriate edits).
I have no idea how to proceed from here...
Go to logs...
Partial log (they're LONG)
docker logs docker-mailman_database_1
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
performing post-bootstrap initialization ... No usable system locales were found.
Use the option "--debug" to see details.
ok
syncing data to disk ... ok
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....LOG: database system was shut down at 2019-04-09 14:37:48 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: autovacuum launcher shutting down
LOG: shutting down
waiting for server to shut down....LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
LOG: database system was shut down at 2019-04-09 14:37:50 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
ERROR: relation "social_auth_usersocialauth" does not exist at character 15
STATEMENT: SELECT 1 from social_auth_usersocialauth
LOG: database system was interrupted; last known up at 2019-04-09 14:48:37 UTC
LOG: database system was not properly shut down; automatic recovery in progress
LOG: invalid record length at 0/1767BB8: wanted 24, got 0
LOG: redo is not required
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: received smart shutdown request
LOG: autovacuum launcher shutting down
LOG: shutting down
LOG: database system is shut down
LOG: database system was shut down at 2019-04-09 15:18:06 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
ERROR: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL: Key (username)=(lists) already exists.
STATEMENT: INSERT INTO "auth_user" ("password", "last_login", "is_superuser", "username", "first_name", "last_name", "email", "is_staff", "is_active", "date_joined") VALUES ('!lS8pgTTV7CyhUbZItku00ZnUGxLQkVEuc3ENZKbV', NULL, true, 'lists', '', '', 'iotic-za(a)reasondigital.co.za', true, true, '2019-04-09T15:39:57.715036+00:00'::timestamptz) RETURNING "auth_user"."id"
TopMemoryContext: 8192 total in 1 blocks; 3480 free (0 chunks); 4712 used
smgr relation table: 24576 total in 2 blocks; 13008 free (4 chunks); 11568 used
WAL record construction: 49768 total in 2 blocks; 6584 free (0 chunks); 43184 used
Checkpointer: 8192 total in 1 blocks; 7992 free (3 chunks); 200 used
PrivateRefCount: 8192 total in 1 blocks; 2840 free (0 chunks); 5352 used
MdSmgr: 8192 total in 1 blocks; 8120 free (0 chunks); 72 used
Pending ops context: 0 total in 0 blocks; 0 free (0 chunks); 0 used
Pending Ops Table: 8192 total in 1 blocks; 2840 free (5 chunks); 5352 used
LOCALLOCK hash: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
Timezones: 104120 total in 2 blocks; 2840 free (0 chunks); 101280 used
ErrorContext: 8192 total in 1 blocks; 8152 free (0 chunks); 40 used
Grand total: 235808 bytes in 13 blocks; 56632 free (12 chunks); 179176 used
ERROR: out of memory
DETAIL: Failed on request of size 4032.
CONTEXT: writing block 0 of relation base/16384/16797
TopMemoryContext: 8192 total in 1 blocks; 3480 free (0 chunks); 4712 used
smgr relation table: 24576 total in 2 blocks; 13008 free (4 chunks); 11568 used
WAL record construction: 49768 total in 2 blocks; 6584 free (0 chunks); 43184 used
Checkpointer: 8192 total in 1 blocks; 7992 free (2 chunks); 200 used
PrivateRefCount: 8192 total in 1 blocks; 2840 free (0 chunks); 5352 used
LOG: could not fork autovacuum worker process: Out of memory
MdSmgr: 8192 total in 1 blocks; 8120 free (0 chunks); 72 used
Pending ops context: 0 total in 0 blocks; 0 free (0 chunks); 0 used
Pending Ops Table: 8192 total in 1 blocks; 2840 free (5 chunks); 5352 used
LOCALLOCK hash: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
LOG: could not fork autovacuum worker process: Out of memory
Timezones: 104120 total in 2 blocks; 2840 free (0 chunks); 101280 used
ErrorContext: 8192 total in 1 blocks; 8152 free (0 chunks); 40 used
Grand total: 235808 bytes in 13 blocks; 56632 free (11 chunks); 179176 used
LOG: could not fork autovacuum worker process: Out of memory
ERROR: out of memory
DETAIL: Failed on request of size 4032.
CONTEXT: writing block 0 of relation base/16384/16797
LOG: could not fork autovacuum worker process: Out of memory
WARNING: could not write block 0 of base/16384/16797
DETAIL: Multiple failures --- write error might be permanent.
TopMemoryContext: 8192 total in 1 blocks; 3480 free (0 chunks); 4712 used
smgr relation table: 24576 total in 2 blocks; 13008 free (4 chunks); 11568 used
WAL record construction: 49768 total in 2 blocks; 6584 free (0 chunks); 43184 used
Checkpointer: 8192 total in 1 blocks; 7992 free (2 chunks); 200 used
PrivateRefCount: 8192 total in 1 blocks; 2840 free (0 chunks); 5352 used
MdSmgr: 8192 total in 1 blocks; 8120 free (0 chunks); 72 used
Pending ops context: 0 total in 0 blocks; 0 free (0 chunks); 0 used
Pending Ops Table: 8192 total in 1 blocks; 2840 free (5 chunks); 5352 used
LOCALLOCK hash: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
TopMemoryContext: 51040 total in 4 blocks; 1344 free (1 chunks); 49696 used
TopTransactionContext: 8192 total in 1 blocks; 6968 free (0 chunks); 1224 used
Timezones: 104120 total in 2 blocks; 2840 free (0 chunks); 101280 used
Statistics snapshot: 0 total in 0 blocks; 0 free (0 chunks); 0 used
ErrorContext: 8192 total in 1 blocks; 8152 free (0 chunks); 40 used
Grand total: 235808 bytes in 13 blocks; 56632 free (11 chunks); 179176 used
Per-database function: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
Per-database table: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
Per-database function: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
Per-database table: 8192 total in 1 blocks; 776 free (0 chunks); 7416 used
Databases hash: 24576 total in 2 blocks; 15072 free (5 chunks); 9504 used
ERROR: out of memory
For UWSGI
cat uwsgi.log
[pid: 49|app: 0|req: 1/1] 172.19.199.1 () {24 vars in 291 bytes} [Tue Apr 9 14:43:43 2019] GET /postorius/lists/ => generated 3710 bytes in 705 msecs (HTTP/1.1 200) 5 headers in 163 bytes (1 switches on core 0)
[pid: 39|app: 0|req: 1/1] 172.19.199.1 () {24 vars in 291 bytes} [Tue Apr 9 15:40:12 2019] GET /postorius/lists/ => generated 3710 bytes in 694 msecs (HTTP/1.1 200) 5 headers in 163 bytes (1 switches on core 0)
[ec2-user@ip-172-31-23-250 logs]$ cat uwsgi-error.log
cat: uwsgi-error.log: Permission denied
[ec2-user@ip-172-31-23-250 logs]$ sudo cat uwsgi-error.log
*** Starting uWSGI 2.0.15 (64bit) on [Tue Apr 9 14:38:03 2019] ***
compiled with version: 6.3.0 on 29 December 2017 00:16:31
os: Linux-4.14.77-70.59.amzn1.x86_64 #1 SMP Mon Nov 12 22:02:45 UTC 2018
nodename: mailman-web
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /opt/mailman-web
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 101
setuid() to 100
chdir() to /opt/mailman-web
your processes number limit is 3860
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:8080 fd 8
uwsgi socket 1 bound to TCP address 0.0.0.0:8000 fd 9
Python version: 2.7.14 (default, Dec 19 2017, 17:52:21) [GCC 6.3.0]
Python main interpreter initialized at 0x55ca1d90e1e0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
[uwsgi-cron] command "./manage.py runjobs minutely" registered as cron task
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs hourly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs daily" registered as cron task
[uwsgi-cron] command "./manage.py runjobs monthly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs weekly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs yearly" registered as cron task
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x55ca1d90e1e0 pid: 1 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 49, cores: 2)
Tue Apr 9 14:38:04 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 51)
[uwsgi-daemons] spawning "./manage.py qcluster" (uid: 100 gid: 101)
14:38:07 [Q] INFO Q Cluster-52 starting.
14:38:07 [Q] INFO Process-1 guarding cluster at 67
14:38:07 [Q] INFO Process-1:1 ready for work at 68
14:38:07 [Q] INFO Process-1:3 pushing tasks at 70
14:38:07 [Q] INFO Q Cluster-52 running.
14:38:07 [Q] INFO Process-1:2 monitoring at 69
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 51 exited after 4 second(s)
Tue Apr 9 14:39:04 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 71)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 71 exited after 269 second(s)
Tue Apr 9 14:43:33 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 79)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 79 exited after 3 second(s)
Tue Apr 9 14:51:35 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 89)
*** Starting uWSGI 2.0.15 (64bit) on [Tue Apr 9 15:39:58 2019] ***
compiled with version: 6.3.0 on 29 December 2017 00:16:31
os: Linux-4.14.109-80.92.amzn1.x86_64 #1 SMP Mon Apr 1 23:07:39 UTC 2019
nodename: mailman-web
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /opt/mailman-web
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 101
setuid() to 100
chdir() to /opt/mailman-web
your processes number limit is 3860
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:8080 fd 8
uwsgi socket 1 bound to TCP address 0.0.0.0:8000 fd 9
Python version: 2.7.14 (default, Dec 19 2017, 17:52:21) [GCC 6.3.0]
Python main interpreter initialized at 0x562e2c0b11e0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
[uwsgi-cron] command "./manage.py runjobs minutely" registered as cron task
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs hourly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs daily" registered as cron task
[uwsgi-cron] command "./manage.py runjobs monthly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs weekly" registered as cron task
[uwsgi-cron] command "./manage.py runjobs yearly" registered as cron task
mapped 166144 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x562e2c0b11e0 pid: 1 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 39, cores: 2)
Tue Apr 9 15:39:59 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 41)
[uwsgi-daemons] spawning "./manage.py qcluster" (uid: 100 gid: 101)
15:40:02 [Q] INFO Q Cluster-43 starting.
15:40:02 [Q] INFO Process-1:1 ready for work at 58
15:40:02 [Q] INFO Process-1:2 monitoring at 59
15:40:02 [Q] INFO Process-1 guarding cluster at 57
15:40:02 [Q] INFO Process-1:3 pushing tasks at 60
15:40:02 [Q] INFO Q Cluster-43 running.
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 41 exited after 4 second(s)
Tue Apr 9 15:40:59 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 63)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 63 exited after 13 second(s)
Tue Apr 9 15:41:59 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 71)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 71 exited after 33 second(s)
Tue Apr 9 15:42:59 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 79)
16:33:36 [Q] ERROR could not translate host name "database" to address: Try again
16:35:17 [Q] INFO Process-1:3 stopped pushing tasks
16:35:25 [Q] ERROR reincarnated pusher Process-1:3 after sudden death
16:35:25 [Q] INFO Process-1:4 pushing tasks at 87
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 79 exited after 3146 second(s)
Tue Apr 9 16:35:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 88)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 88 exited after 3 second(s)
Tue Apr 9 16:36:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 96)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 96 exited after 3 second(s)
Tue Apr 9 16:37:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 104)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 104 exited after 3 second(s)
Tue Apr 9 16:38:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 112)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 112 exited after 3 second(s)
Tue Apr 9 16:39:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 120)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 120 exited after 3 second(s)
Tue Apr 9 16:40:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 128)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 128 exited after 3 second(s)
Tue Apr 9 16:41:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 136)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 136 exited after 3 second(s)
Tue Apr 9 16:42:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 144)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 144 exited after 3 second(s)
Tue Apr 9 16:43:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 152)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 152 exited after 3 second(s)
Tue Apr 9 16:44:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 160)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 160 exited after 3 second(s)
Tue Apr 9 16:45:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 168)
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" running with pid 168 exited after 2 second(s)
Tue Apr 9 16:45:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 176)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 176 exited after 3 second(s)
Tue Apr 9 16:46:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 184)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 184 exited after 2 second(s)
Tue Apr 9 16:47:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 192)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 192 exited after 3 second(s)
Tue Apr 9 16:48:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 200)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 200 exited after 3 second(s)
Tue Apr 9 16:49:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 208)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 208 exited after 3 second(s)
Tue Apr 9 16:50:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 216)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 216 exited after 3 second(s)
Tue Apr 9 16:51:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 224)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 224 exited after 3 second(s)
Tue Apr 9 16:52:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 232)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 232 exited after 3 second(s)
Tue Apr 9 16:53:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 240)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 240 exited after 2 second(s)
Tue Apr 9 16:54:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 248)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 248 exited after 2 second(s)
Tue Apr 9 16:55:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 256)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 256 exited after 3 second(s)
Tue Apr 9 16:56:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 264)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 264 exited after 3 second(s)
Tue Apr 9 16:57:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 272)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 272 exited after 3 second(s)
Tue Apr 9 16:58:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 280)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 280 exited after 3 second(s)
Tue Apr 9 16:59:25 2019 - [uwsgi-cron] running "./manage.py runjobs minutely" (pid 288)
[uwsgi-cron] command "./manage.py runjobs minutely" running with pid 288 exited after 3 second(s)
Tue Apr 9 17:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs quarter_hourly" (pid 296)
Tue Apr 9 17:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs hourly" (pid 297)
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
Traceback (most recent call last):
File "./manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 337, in execute
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 337, in execute
django.setup()
django.setup()
File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
File "/usr/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
apps.populate(settings.INSTALLED_APPS)
File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
File "/usr/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
app_config.import_models()
app_config.import_models()
File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
File "/usr/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
self.models_module = import_module(models_module_name)
self.models_module = import_module(models_module_name)
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
__import__(name)
File "/usr/local/lib/python2.7/site-packages/hyperkitty/models/__init__.py", line 28, in <module>
File "/usr/local/lib/python2.7/site-packages/hyperkitty/models/__init__.py", line 28, in <module>
from .email import Email, Attachment
from .email import Email, Attachment
File "/usr/local/lib/python2.7/site-packages/hyperkitty/models/email.py", line 37, in <module>
File "/usr/local/lib/python2.7/site-packages/hyperkitty/models/email.py", line 37, in <module>
from hyperkitty.lib.analysis import compute_thread_order_and_depth
from hyperkitty.lib.analysis import compute_thread_order_and_depth
File "/usr/local/lib/python2.7/site-packages/hyperkitty/lib/analysis.py", line 28, in <module>
File "/usr/local/lib/python2.7/site-packages/hyperkitty/lib/analysis.py", line 28, in <module>
import networkx as nx
import networkx as nx
File "/usr/local/lib/python2.7/site-packages/networkx/__init__.py", line 114, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/__init__.py", line 114, in <module>
import networkx.generators
import networkx.generators
File "/usr/local/lib/python2.7/site-packages/networkx/generators/__init__.py", line 6, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/generators/__init__.py", line 6, in <module>
from networkx.generators.classic import *
from networkx.generators.classic import *
File "/usr/local/lib/python2.7/site-packages/networkx/generators/classic.py", line 26, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/generators/classic.py", line 26, in <module>
from networkx.algorithms.bipartite.generators import complete_bipartite_graph
from networkx.algorithms.bipartite.generators import complete_bipartite_graph
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/__init__.py", line 57, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/__init__.py", line 57, in <module>
import networkx.algorithms.connectivity
import networkx.algorithms.connectivity
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/connectivity/__init__.py", line 3, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/connectivity/__init__.py", line 3, in <module>
from .connectivity import *
from .connectivity import *
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/connectivity/connectivity.py", line 13, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/connectivity/connectivity.py", line 13, in <module>
from networkx.algorithms.flow import boykov_kolmogorov
from networkx.algorithms.flow import boykov_kolmogorov
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/flow/__init__.py", line 1, in <module>
File "/usr/local/lib/python2.7/site-packages/networkx/algorithms/flow/__init__.py", line 1, in <module>
from .maxflow import *
from .maxflow import *
MemoryErrorMemoryError
18:16:37 [Q] ERROR could not translate host name "database" to address: Try again
...etc ...
For Mailman
cat mailmanweb.log
ERROR 2019-04-09 20:05:48,878 934 hyperkitty.lib.utils Failed to update the fulltext index: could not translate host name "database" to address: Try again
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/hyperkitty/lib/utils.py", line 186, in run_with_lock
fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/hyperkitty/search_indexes.py", line 87, in update_index
update_cmd.update_backend("hyperkitty", "default")
File "/usr/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 240, in update_backend
total = qs.count()
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 363, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 498, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 479, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 874, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
return self._cursor()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
self.ensure_connection()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
OperationalError: could not translate host name "database" to address: Try again
ERROR 2019-04-09 20:15:08,192 983 hyperkitty.lib.utils Failed to update the fulltext index: could not translate host name "database" to address: Try again
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/hyperkitty/lib/utils.py", line 186, in run_with_lock
fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/hyperkitty/search_indexes.py", line 87, in update_index
update_cmd.update_backend("hyperkitty", "default")
File "/usr/local/lib/python2.7/site-packages/haystack/management/commands/update_index.py", line 240, in update_backend
total = qs.count()
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 363, in count
return self.query.get_count(using=self.db)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 498, in get_count
number = obj.get_aggregation(using, ['__count'])['__count']
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 479, in get_aggregation
result = compiler.execute_sql(SINGLE)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 874, in execute_sql
cursor = self.connection.cursor()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254, in cursor
return self._cursor()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 229, in _cursor
self.ensure_connection()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 213, in ensure_connection
self.connect()
File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/base.py", line 189, in connect
self.connection = self.get_new_connection(conn_params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
connection = Database.connect(**conn_params)
File "/usr/local/lib/python2.7/site-packages/psycopg2/__init__.py", line 130, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
OperationalError: could not translate host name "database" to address: Try again
6 years, 6 months
Re: Importing mbox files into archive defect with lines with From
by Alex Schuilenburg
Hi Stephen
On 19/09/2023 19:59, Stephen J. Turnbull wrote:
> Did you intend to send this to me only?
Nope, apologies. I'm happy to add the list back in - I thought I hit
reply-to-list.
> Alex Schuilenburg writes:
>
> > Yes, I retested with from the git head as well. Same result. It
> > sees the line
> >
> > From 1st April 2019, our turnover has exceeded the threshold for
> >
> > as the start of a new message and reports no anomolies although
> > there is clearly one.
>
> If there is an empty line preceding that one, and the "F" is in the
> first column (no marginal whitespace), that is considered correct
> behavior for processing an mbox file.
As an email separator agreed, provided the body has all "First "
(preceded by a blank line and nothing preceding on the same line)
suitably escaped.
> If you don't like it, don't use the mbox format. The maildir
> message-per-file format is widely supported, and if you want a
> folder-per-file rather than message-per-file format, MMDF is fairly
> widely supported, and does not suffer from the ambiguities of mbox.
> All three (with variations) are supported by recent Python 3.
Understood. My issue is the lists were imported from MM2.1 in 2020 into
MM3.2.1 onto hyperkitty+mariadb/mysql.
So the current installation has no mbox files - archives are stored in a
mysql database.
> > The old MM3 installation I am migrating away from is obviously
> > broken in that its mbox archive when downloaded is not escaped,
>
> That's interesting. I presume that's because HyperKitty saves
> messages in a database rather than as mbox files, and the old mbox
> exporter just flattened and concatenated the messages. Or perhaps
> that also uses mailbox.mbox and old enough versions of that didn't
> From-stuff.
Spot on.
I have to move the lists onto a new Debian 12 server using the native
mailman 3.3.8 & mailman-web 0+20200530-2 packages. I tried dropping in
the old mailman3 database under the new software but that did not work.
Instead I manually imported the old mailman3 data directly into the new
mailman3 database as, after inspections, there were no new tables and
only a couple of additional fields which have suitable defaults. So
dumped the mailman3 data with
mysqldump --no-create-info --no-create-db --disable-keys
--complete-insert --ignore-table=mailman3.alembic_version mailman3
and simply imported the dump into the new schema. OK so far. The lists
showed up in postorius, obviously without archives in hyperkitty.
> The normal way to upgrade a HyperKitty archive is to do nothing, just
> upgrade the software. I guess you moved to a new host and deleted the
> database? The preferred way is to dump the database to SQL, and then
> load it in to the new database directly rather than downloading the
> mbox files and importing. No ambiguity and much faster.
Thats what I thought initially, but that failed as per
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message….
As my old installation appears to have the django_migrations table
inconsistent with the state of the database, and Debian have been
unresponsive so far.
Unfortunately the same mailman3-style manual import of mailman3web from
old mailman3web db into the new db was not possible. There were
additional tables and fields that needed values rather than defaults.
Instead I opted to download the mbox exports from the old installation
via the web interface and import them into the new installation for
expediency.This is where I was when I posted the message.
However, the old installation's mbox archive export (from the database)
was problematic (e.g. 5min timeout) but after getting around those
issues I ended up with some broken mbox files (e.g. the "From 1st April
..." line within a message, resulting in hyperkitty_import failures).
> > although I tested downloading this thread archive and it is clearly
> > escaped. Unfortunately I cannot tell if hyperkitty_import works
> > using the git head since I am restricted to using the version
> > provided with Debian 12.
>
> In July it worked for me on a site with 5300 archived lists. I assume
> some of them had From-stuffed lines. Can't be sure, most of the posts
> are machine-generated, but lines beginning with "From " are pretty
> common in natural English.
I'd be surprised if there were not any. I have 2003 lists and several
occurrences.
>
> > There are indeed two versions though of mailbox.py on the VM:
> >
> > /usr/lib/python3/dist-packages/mailman/utilities/mailbox.py: from
> > mailman3 (3.3.8-2~deb12u1)
> > /usr/lib/python3.11/mailbox.py from libpython3.11-stdlib (3.11.2-6)
>
> The first applies one small change to the second (a so-called
> "monkey-patch") for use by Mailman core. It should only be visible
> to Python by the name "mailman.utilities.mailbox". The HyperKitty
> utilities import the name "mailbox". If overwriting the first with
> the second changes HyperKitty's behavior, something is wrong with
> sys.path.
Then I guess that is the case in Debian 12.
>
> > the latter of which appears far more substantial , so I dropped
> > that module over the former and reran hyperkitty_import over the
> > '>' escaped mbox, and while it did import without error, it did
> > leave the escape in place (i.e. the 'From ...' was quoted when
> > viewed).
>
> This is considered correct behavior. It is not possible to determine
> whether the escape was in the original, or added by a receiving MTA.
> Better to leave it.
I thought that ">From " would be escaped to ">>From ", and so on, so the
escape could easily be reversed when imported. I tested exports from the
mailman-users lists and lines beginning "From " (preceded by a blank
line) are escaped to ">From ", so incorrectly figured this would be
unescaped by hyperkitty_import. After all, I would expect that an export
of the archive to mbox, followed by a delete of the archive, followed by
a hyperkitty_import of the archive, should leave you at the same place.
Not with ">From " escapes in the new archives. In fact I also had a
number of messages with "Message-ID: <>" and worse: all messages with
attachments had the text/plain content empty.
So mbox exports from MM 3.2.1 on Debian 10 (using hyperkitty+mysql) are
broken.
>
> > The unescaped mbox import died in the same way.
>
> As expected.
>
> > Anyhow, thanks for your suggestion. For now I can stick with a
> > manual repair and spaced escape of From_.
>
> If the old database is still available, I recommend dumping that and
> loading it into a fresh version of the DBMS.
> ...
Thanks for the pointer. As I had already done the same with mailman3,
so repeated the excercise. The following dump and import worked.
oldhost> mysqldump --no-create-info --no-create-db --disable-keys
--complete-insert mailman3web > mailman3web.sql
newhost> mysql
MariaDB [(none]> use mailman3web
MariaDB [mailman3web]> source mailman3web.sql
Thanks again
-- Alex
2 years
Re: config incoming email in my domain cpanel
by tlhackque
On 03-Aug-17 17:14, Rafael Mora wrote:
>
>
> El jue., 3 ago. 2017 a las 16:07, tlhackque via Mailman-users
> (<mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>>)
> escribió:
>
> You've said you're working with labbrands.com <http://labbrands.com>.
>
> labbrands.com <http://labbrands.com> has address 192.185.51.89
> mail.labbrands.com <http://mail.labbrands.com> has address
> 192.185.51.89
> www.labbrands.com <http://www.labbrands.com> is an alias for
> labbrands.com <http://labbrands.com>.
>
> telnet proves that Exim is running on that IP address. Either you're
> running it, or hostgator is redirecting the SMTP ports.
>
> You added:
> lists.labbrands.com <http://lists.labbrands.com> has address
> 190.145.27.66
>
> Presumably that's a different machine.
>
> So that now needs an MX record, and a firewall routing.
>
>
> I added the MX record on our hostgator hosting like this:
> image.png
>
> Is it correct???
>
No. And sending images to the mailing list doesn't get you help.
You set up lists.labbrands.com as a secondary server, not an independent
mail domain. This will cause you serious grief.
You need to get some help from a more suitable list. Setting this up is
not mailman-specific, and getting it right is more than a quick note.
Try the Postfix documentation & mailing lists.
>
>
> And the SPF
> records ... and spam and virus filters, and all the other stuff
> you need
> to setup an independent, public-facing smtp server.
>
> But at least you won't have to relay.
>
> Good luck.
>
> On 03-Aug-17 16:55, Rafael Mora wrote:
> > El jue., 3 ago. 2017 a las 15:49, tlhackque via Mailman-users (<
> > mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>>)
> escribió:
> >
> >> I think you're confused.
> >>
> >> You already have mail.labbrands.com <http://mail.labbrands.com>
> set up as the MX record for
> >> labbrands.com <http://labbrands.com>. And it has an A record
> with the same address as your
> >> webserver.
> >>
> > I'm working with the hostgator mailserver, we are not running a
> local
> > mailserver.
> >
> > As suggested I added an A record like this:
> > [image: image.png]
> >
> > Is it correct? is it redirecting to my Ip so I can redirect it to my
> > postfix/mm3 server?
> >
> >
> >> So if you're getting e-mail on that domain, there's another e-mail
> >> server running on that IP address. You can't have 2 servers
> on port
> >> 25. In that case, as has been noted before, you'll need to setup a
> >> relay in that server, not a firewall redirect. Depending on
> your MTA,
> >> you would need to relay to your internal server. And make sure
> your
> >> firewall setup allows your MTA to do this.
> >>
> >> We can see it's EXIM:
> >>
> >> telnet mail.labbrands.com <http://mail.labbrands.com> 25
> >> Trying 192.185.51.89...
> >> Connected to mail.labbrands.com <http://mail.labbrands.com>
> (192.185.51.89).
> >> Escape character is '^]'.
> >> help
> >> 220-gator4137.hostgator.com
> <http://220-gator4137.hostgator.com> ESMTP Exim 4.87 #1 Thu, 03
> Aug 2017 15:47:48
> >> -0500
> >> 220-We do not authorize the use of this system to transport
> unsolicited,
> >> 220 and/or bulk e-mail.
> >> 214-Commands supported:
> >> 214 AUTH STARTTLS HELO EHLO MAIL RCPT DATA NOOP QUIT RSET HELP
> >> quit
> >> 221 gator4137.hostgator.com <http://gator4137.hostgator.com>
> closing connection
> >> Connection closed by foreign host.
> >>
> >> Or, consolidate all your e-mail to one server - which is a lot
> easier to
> >> manage unless you have a really big operation. Postfix is
> probably the
> >> right choice, but requires more setup.
> >>
> >> Although Mailman3 configuration is not well documented (as you've
> >> discovered), you may want to get help from someone with more
> general
> >> network and mail experience. You're now into territory that is, as
> >> Simon indicated, not Mailman-specific.
> >>
> >> On 03-Aug-17 16:28, Rafael Mora wrote:
> >>> El jue., 3 ago. 2017 a las 15:26, Mark Sapiro
> (<mark(a)msapiro.net <mailto:mark@msapiro.net>>)
> >> escribió:
> >>>> On 08/03/2017 01:22 PM, Rafael Mora wrote:
> >>>>> El jue., 3 ago. 2017 a las 15:18, Mark Sapiro
> (<mark(a)msapiro.net <mailto:mark@msapiro.net>>)
> >>>> escribió:
> >>>>>> You need to forward port 25 for SMTP mail delivery and if
> you want the
> >>>>>> web UI (Postorius and HyperKitty) accessible from the
> outside, port 80
> >>>>>> for http and/or port 443 for https
> >>>>>> <
> >>>>>>
> >>
> https://www.iana.org/assignments/service-names-port-numbers/service-names-p…
> >>>>>>> .
> >>>>> I mean for incoming mail redirected from my hostgator
> hosting, because
> >>>> when
> >>>>> I suscribe an email address MM3 sends a confirmation email,
> so I have
> >> to
> >>>>> reply to be suscribed to the list.
> >>>> As I said, for mail delivery you need to forward port 25 to
> the Mailman
> >>>> server.
> >>>>
> >>> Ok so I'll redirect in my zentyal firewall the port 25 to my local
> >> centos7
> >>> with postfix and MM3 with IP 192.168.1.42. Thanks Mark.
> >>>
> >>>
> >>>> --
> >>>> Mark Sapiro <mark(a)msapiro.net <mailto:mark@msapiro.net>>
> The highway is for gamblers,
> >>>> San Francisco Bay Area, California better use your sense -
> B. Dylan
> >>>> _______________________________________________
> >>>> Mailman-users mailing list
> >>>> mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>
> >>>>
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >>>>
> >> --
> >> This communication may not represent my employer's views,
> >> if any, on the matters discussed.
> >>
> >> _______________________________________________
> >> Mailman-users mailing list
> >> mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>
> >>
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >>
>
> --
> This communication may not represent my employer's views,
> if any, on the matters discussed.
>
> _______________________________________________
> Mailman-users mailing list
> mailman-users(a)mailman3.org <mailto:mailman-users@mailman3.org>
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>
> --
>
> Atentamente / Best Regards
>
> Ing. Rafael Mora
>
--
This communication may not represent my employer's views,
if any, on the matters discussed.
8 years, 2 months
Re: Member Issue Discovered
by Brian Carpenter
On 10/19/20 11:32 PM, Mark Sapiro wrote:
> On 10/19/20 7:42 PM, Brian Carpenter wrote:
>> On 10/19/20 10:23 PM, Mark Sapiro wrote:
>>> And the import21 created the user and address records for this user.
>> Does the same thing for a new subscriber as well. So there is no pathway
>> to change a real name that is associated with an email address. None.
>> Zilch. Mailman 2 made this so easy and Mailman 3 made it impossible. I
>> will let someone else file the bug report. You don't really think that
>> this is an issue which means it will be years before it is addressed. So
>> I will save me some time. I will learn to live with it.
>
> I didn't say I didn't think it was a real issue. I mostly questioned
> whether a new subscription should change an existing display_name. I
> agree that there should be a way for a user to change the display_name
> associated with her address. I'm not so sure about a list admin.
Respectively, I think you are asking the wrong question here. The real
question is why isn't a display_name being removed when a list
subscriber is unsubscribed. Also list members being forced into a
powerless role by Mailman 3's architecture have no way of changing a
name except through a list owner unless they register an account but
then again, *even that doesn't allow them to change their display name*.
There are also other use cases in which a list owner will want to
identify portions of their membership roster through the use of the
display_name.
Most list members interact with a Mailing list program via email. I know
you understand that. However it is List owners that are the "real" power
users of Mailman and so the admin interface should be designed in such a
way that empowers List owners. Taking the ability away from List owners
(and List members to make), what should be a simple name change, weakens
them.
> And in any case, I'm only one person. I'm not the only one deciding
> what's important and what's not. I only decide what I want to work on,
> not what anyone else thinks is important or wants to work on, so even if
> I think something is not worth doing, that doesn't mean it won't get
> done, and once again for emphasis, I do thing a user should be able to
> change the display name associated with her address(es).
Again I think we are missing something here from this conversation: data
is not being removed. A list subscriber being removed (unsubscribed)
should have their information totally removed. It is not. I now have one
list owner who realizes this and it is not good. How does this not
violate GDPR? This is why I think all the mailman developers ought to
make this issue a high priority. It is creepy to see a name returned out
of nowhere when someone resubscribes to a list without associating a
name with a second resubscribing. I think this puts List owners at a
disadvantage.
>
> Mailman 3 is totally different from Mailman 2.1 in this respect. Mailman
> 2.1 had no concept of user. All it knew was addresses subscribed to
> lists and an address subscribed to one list had no connection to the
> same address subscribed to another list or being an owner or moderator
> of a list.
I think this should still be the case for non-registered list members.
List owners/moderators are different. They HAVE to be registered users
of Postorius/Affinity in order to manage/moderate a list. I wonder what
the majority of list members' scenarios are. Is the majority scenario in
play today the one where most list members are only subscribed to a
single list on a single mailman instance? Or is it the scenario that is
in majority use the one where a single list member is subscribed to
multiple lists? I think that it is important to find out because it
should govern the development process to a certain degree. This single
approach of having registered user account with multiple associated
email accounts elevates one scenario at the expense of the other. It
also assumes/requires that list members have a registered user account
and I can tell you from my experience that is not going to happen. The
majority of mailman 2 users will not registered with a Mailman 3 admin
interface. Some will of course, the majority will not. Maybe the
behavior should and will change but it will take years for that to
happen. So what I am challenging and questioning here is the approach
where such assumptions are being made already that results in the
weakening of list owners.
>
> Mailman 3 does have a concept of user and addresses belonging to a user.
> This complicates things in some ways. In Mailman 2.1 we could have "The
> Boss <user(a)example.com" as a member of one list and "Just a Peon
> <user(a)example.com>" as a member of another list. In Mailman 3 that is
> not possible unless the addresses are tweaked in some way to make them
> different.
Perhaps there is a better approach here to handle single member
w/multiple subscriptions that doesn't hurt the ability of List owners to
serve the needs of list members who do not have a registered user
account and prevents the removal of data when a SINGLE user unsubscribes
from a list.
>
> You don't seem to be concerned about the case where I subscribe to a
> second list with a different display name and am surprised to find my
> display name changed on the first list, but it's something that I have
> to consider.
I am not concerned because I think such cases are in the extreme
minority. Are you saying that people are now going to be using different
real names associated with their other subscriptions as well and that is
going to be so needed it warrants this complex system that you have come
up with? If someone whats to have a different name associated with a
different subscription/email address then I am going to tell them create
a new user account and use a good password management program to keep
track of their logins. Right now I think this minority use case scenario
is negatively impacting other administrative tasks of Mailman,
particularly in the removal of user data when someone unsubscribes from
a list. Its the use of a square peg being forced into a round hole.
Except in this case most of the holes are round but by golly, we are
going to use that square peg regardless!
>
> As far as filing or not filing an issue, issues in the GitLab trackers
> are how we track these things. Threads on mailing lists are appropriate
> for discussion of issues, but if something is going to get changed or
> fixed, an issue in the tracker is the way to ensure it doesn't get put
> aside and forgotten. If this is as important to you as it seems to be,
> please file the issue.
I will but what is disturbing is that you don't think that the
non-removal of user data in cases of unsubscribing is not important or
even an issue.
--
Brian Carpenter
Harmonylists.com
Emwd.com
4 years, 11 months