Re: Missing Base64 encoding from control message reply
by Mark Sapiro
On 9/22/23 09:32, Stephen J. Turnbull wrote:
>
> To diagnose, we really need to see that in context of the raw message
> as received by the user.
It was posted. The issue is the message declares
Content-Transfer-Encoding: base64
but the body is not base64 encoded so the MUA tries to base64 decode it
and even though that's invalid, produces the garbled output.
> The only other thing I can think of is it might have something to do
> with your internationalization settings. What is the default language
> of the list? (This is available in Postorius.) Have you changed the
> default encoding for English on your site? (I think you need to look
> in mailman.cfg for this.) What is the default language of the site?
I note that the message also declares
Content-Type: text/plain; charset="utf-8"
and Python's email package will base64 encode utf-8 message bodies. I
also note that the messages Subject: is RFC 2047 encoded as utf-8. These
tell me that the user's preferred_language if set or the list's
preferred_language is not English or the charset for English is changed
to utf-8.
I tried to duplicate this in a test installation with the charset for
English set to utf-8, but I get a similar response but with
Subject: =?utf-8?q?The_results_of_your_email_commands?=
and
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
2 years, 2 months
Docker Mailman3 installation frustration
by ccsmith＠cetsi.com
I've been working on installing Mailman3 for several weeks now, with limited success. I have tried various installation methods following instructions on numerous different web pages. My experience has been that the instructions are often conflicting and never complete.
My best success has been with the Docker version, so that's what I'm going to concentrate on here.
I'm running Ubuntu 16.04 (under Citrix XenServer), with Apache httpd and postfix.
Following the instructions on https://asynchronous.in/docker-mailman/, I did the following:
* created the directories "/opt/mailman/core" and "/opt/mailman/web",
* created "/opt/mailman/core/mailman-extra.cfg" and inserted the suggested [mta] directives,
* inserted the "transport_maps", "local-recipient_maps", and "relay_domains" entries in the /etc/postfix/main.cf file,
* created the necessary apache config files,
* cloned "https://github.com/maxking/docker-mailman",
* ran "docker-compose up -d"
At this point, I am able to connect with Postorius. All of the text and icons show up properly; all is well. It tells me (correctly) that "There are currently no mailing lists."
When I click on the "Login" link at the top right, the "Sign In" page displays correctly. (Aside: Why do so many apps/sites have difficulty with choosing between "login" and "sign in" and, in fact, use both terms? It can really be off-putting to non-tech users... and me.) If I input an incorrect login, I receive (correctly) the message "The username and/or password you specified are not correct." However, if I input the correct sign-in, I receive: "Server Error (500)".
Running "sudo docker exec -it mailman-web cat ../mailman-web-data/logs/mailmanweb.log" shows me:
ERROR 2017-07-12 20:34:25,073 25 django.request Internal Server Error: /accounts/login/
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 39, in inner
response = get_response(request)
... more stuff ...
File "/usr/local/lib/python2.7/smtplib.py", line 747, in sendmail
raise SMTPRecipientsRefused(senderrs)
SMTPRecipientsRefused: {u'pacetechadmin(a)cetsi.com': (451, '4.3.0 <pacetechadmin(a)cetsi.com>: Temporary lookup failure')}
On the Docker host (ubuntuServer2), /var/log/mail.log shows:
Jul 12 13:48:53 ubuntuServer2 postfix/smtpd[19931]: error: open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:53 ubuntuServer2 postfix/smtpd[19931]: error: open /opt/mailman/core/var/data/postfix_lmtp: No such file or directory
Jul 12 13:48:53 ubuntuServer2 postfix/smtpd[19931]: connect from unknown[172.19.199.3]
Jul 12 13:48:53 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: regexp:/opt/mailman/core/var/data/postfix_domains is unavailable. open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:53 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: relay_domains: regexp:/opt/mailman/core/var/data/postfix_domains: table lookup problem
Jul 12 13:48:53 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: relay_domains lookup failure
Jul 12 13:48:53 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: regexp:/opt/mailman/core/var/data/postfix_domains is unavailable. open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:53 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: relay_domains: regexp:/opt/mailman/core/var/data/postfix_domains: table lookup problem
Jul 12 13:48:53 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: relay_domains lookup failure
Jul 12 13:48:53 ubuntuServer2 postfix/smtpd[19931]: NOQUEUE: reject: RCPT from unknown[172.19.199.3]: 451 4.3.0 <pacetechadmin(a)cetsi.com>: Temporary lookup failure; from=<postorius(a)localhost.local> to=<pacetechadmin(a)cetsi.com> proto=ESMTP helo=<mailman-web>
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: error: open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: error: open /opt/mailman/core/var/data/postfix_lmtp: No such file or directory
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: connect from unknown[172.19.199.3]
Jul 12 13:48:54 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: regexp:/opt/mailman/core/var/data/postfix_domains is unavailable. open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:54 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: relay_domains: regexp:/opt/mailman/core/var/data/postfix_domains: table lookup problem
Jul 12 13:48:54 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: relay_domains lookup failure
Jul 12 13:48:54 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: regexp:/opt/mailman/core/var/data/postfix_lmtp is unavailable. open /opt/mailman/core/var/data/postfix_lmtp: No such file or directory
Jul 12 13:48:54 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: regexp:/opt/mailman/core/var/data/postfix_lmtp lookup error for "root@localhost"
Jul 12 13:48:54 ubuntuServer2 postfix/trivial-rewrite[19843]: warning: transport_maps lookup failure
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: NOQUEUE: reject: RCPT from unknown[172.19.199.3]: 451 4.3.0 <root@localhost>: Temporary lookup failure; from=<root(a)localhost.local> to=<root@localhost> proto=ESMTP helo=<mailman-web>
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: disconnect from unknown[172.19.199.3] ehlo=1 mail=1 rcpt=0/1 rset=1 quit=1 commands=4/5
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19931]: lost connection after RSET from unknown[172.19.199.3]
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19931]: disconnect from unknown[172.19.199.3] ehlo=1 mail=1 rcpt=0/1 rset=1 commands=3/4
And /var/log/mail.err is filling up with:
Jul 12 13:48:53 ubuntuServer2 postfix/smtpd[19931]: error: open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:53 ubuntuServer2 postfix/smtpd[19931]: error: open /opt/mailman/core/var/data/postfix_lmtp: No such file or directory
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: error: open /opt/mailman/core/var/data/postfix_domains: No such file or directory
Jul 12 13:48:54 ubuntuServer2 postfix/smtpd[19932]: error: open /opt/mailman/core/var/data/postfix_lmtp: No such file or directory
Any thoughts on what the problem is? Have I missed a step?
8 years, 4 months
Re: Getting rid of held messages from the command line
by Mark Sapiro
On 03/14/2018 03:04 AM, Aymeric Mansoux wrote:
>
> So far so good, however, for some reasons, a couple of our lists have started to accumulate held messages that cannot be manipulated via the REST API. Regardless if we use the web interface or mailmanclient, it ends up with:
>
> Mar 14 10:58:32 2018 (24287) REST request handler error:
> Traceback (most recent call last):
> File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run
> self.result = application(self.environ, self.start_response)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 50, in wrapper
> rtn = function(*args, **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/wsgiapp.py", line 214, in __call__
> return super().__call__(environ, start_response)
> File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
> responder(req, resp, **params)
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 167, in on_get
> resource = self._make_collection(request)
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/helpers.py", line 159, in _make_collection
> for resource in collection]
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/helpers.py", line 159, in <listcomp>
> for resource in collection]
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 157, in _resource_as_dict
> resource = self._make_resource(request.id)
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 70, in _make_resource
> resource = super()._make_resource(request_id)
> File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 39, in _make_resource
> results = requests.get_request(request_id)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request
> result.data_hash, expunge=False)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm
> value = json.loads(keyvalue.value)
> File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
> obj, end = self.scan_once(s, idx)
> json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
> Mar 14 10:58:32 2018 (24287) 127.0.0.1 - - "GET /3.1/lists/xx(a)xx.xx/held HTTP/1.1" 500 59
>
> Is there a way to manipulate/see/remove these mails differently? Right now this prevent moderation of any held messages (note: some lists do not suffer from this and the held panel or mailmanclient works perfectly fine).
We've seen this issue before with a different underlying cause. See
<https://gitlab.com/mailman/mailman/issues/256>.
There are a couple of problems here. The serious problem is if there is
a held message which somehow throws an exception in processing in
mailman/rest/post_moderation.py, that totally breaks the Postorius held
messages view for that list. We need to fix that.
Also it would be good to see the problem message.
Here are some things you can do in 'mailman shell' (aka mailman
withlist). It is not clear what might not work, but you can try
mailman shell -l list@domain
which will respond
Welcome to the GNU Mailman shell
The variable 'm' is the list@domain mailing list
>>>
Then you should be able to do at the >>> prompts:
>>> req_db = IListRequests(m)
>>> reqs = list(req_db.held_requests)
reqs is now a list of the held requests and you can do something like
>>> for req in reqs:
... print('{}: {}'.format(req.id, req.request_type))
...
or since req_db.held_requests is a generator
>>> for req in rdb.held_requests:
... print('{}: {}'.format(req.id, req.request_type))
...
either of which will produce a list like
1: RequestType.held_message
with the IDs and types of each request.
You can handle these requests with something like
>>> from mailman.app.moderator import handle_message
>>> handle_message(m, 1, Action.defer)
where the second argument is the ID of the message and the action is one
of Action.defer, Action.hold, Action.discard, Action.reject or
Action.accept, the first two of which do nothing, but the others will do
the indicated action.
You can see the actual messages like this
>>> msg_db = getUtility(IMessageStore)
then
>>> key = req[0].key
>>> msg_db.get_message_by_id(key)
which will return the actual held message that is in this case index 0
in the list of held_requests.
You may have to do
>>> commit()
when done, although if you don't do abort(), just exiting with control-D
should do a commit().
I hope you find this helpful, and if you do determine the content of the
offending message(s), it will be helpful if you post that.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
7 years, 8 months
Re: 404 on fresh install (Debian11)
by Mark Sapiro
On 3/22/23 03:42, thomas--- via Mailman-users wrote:
> Hello,
> very new to mailman3, I used to manage mailman2 on a Debian 10 server.
> The server has been upgraded to Debian 11 and mailman2 disappear…
>
> I try fresh install of mailman3, prerequisite to data migration, but without success.
> If some can help, I would be really greatful.
>
> <VirtualHost *:80>
> ServerName newsletter.mydomain.tld
>
> Alias /static /var/lib/mailman3/web/static
> Alias /favicon.ico /var/lib/mailman3/web/static/postorius/img/favicon.ico
>
> DocumentRoot /var/lib/mailman3/web/static
> <Directory "/var/lib/mailman3/web/static">
> Require all granted
You can probably fix the below error by adding
options +Indexes
here, but see below.
> And now I have a 404 answer when consulting my http page.
>
> When trying to access `/static` sub-path, I get `Cannot serve directory /var/lib/mailman3/web/static/admin/: No matching DirectoryIndex found`.
It is unclear to me what is doing a GET of /static/admin/ and why. GETs
should by of actual files, e.g., things like /static/admin/css/base.css
> I am a bit lost with that.
So are we. Questions about the Debian package are better asked of
Debian. See https://wiki.list.org/x/12812344 and
https://mail.python.org/archives/list/mailman-users@python.org/message/T6LB…
We can provide much better help and support if you install via the
resources mentioned in the reply at
https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
2 years, 8 months
Re: Column 'starting_email_id' cannot be null
by Odhiambo Washington
On Mon, Aug 18, 2025 at 11:32 AM kroerig--- via Mailman-users <
mailman-users(a)mailman3.org> wrote:
> Hello,
>
> after upgrading from Debian 11 to Debian 12 achiving incoming mails always
> fails with
>
> Traceback (most recent call last):
> File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
> 581, in get_or_create
> return self.get(**kwargs), False
> File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
> 435, in get
> raise self.model.DoesNotExist(
>
> During handling of the above exception (Thread matching query does not
> exist.), another exception occurred:
> File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py",
> line 73, in execute
> return self.cursor.execute(query, args)
> File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 209, in
> execute
> res = self._query(query)
> File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 316, in
> _query
> self._do_get_result(db)
> File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 147, in
> _do_get_result
> self._result = result = self._get_result()
> File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 349, in
> _get_result
> return self._get_db().store_result()
>
> During handling of the above exception ((1048, "Column 'starting_email_id'
> cannot be null")), another exception occurred:
> File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py",
> line 47, in inner
> response = get_response(request)
> File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line
> 181, in _get_response
> response = wrapped_callback(request, *callback_args, **callback_kwargs)
> File "/usr/lib/python3/dist-packages/django/views/decorators/http.py",
> line 40, in inner
> return func(request, *args, **kwargs)
> File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py", line
> 117, in _decorator
> return func(request, *args, **kwargs)
> File "/usr/lib/python3/dist-packages/django/views/decorators/csrf.py",
> line 54, in wrapped_view
> return view_func(*args, **kwargs)
> File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py", line
> 163, in archive
> add_to_list(mlist_fqdn, msg)
> File "/usr/lib/python3/dist-packages/hyperkitty/lib/incoming.py", line
> 153, in add_to_list
> email.save()
> File "/usr/lib/python3/dist-packages/django/db/models/base.py", line
> 739, in save
> self.save_base(using=using, force_insert=force_insert,
> File "/usr/lib/python3/dist-packages/django/db/models/base.py", line
> 763, in save_base
> pre_save.send(
> File "/usr/lib/python3/dist-packages/django/dispatch/dispatcher.py",
> line 180, in send
> return [
> File "/usr/lib/python3/dist-packages/django/dispatch/dispatcher.py",
> line 181, in <listcomp>
> (receiver, receiver(signal=self, sender=sender, **named))
> File "/usr/lib/python3/dist-packages/hyperkitty/signals.py", line 48, in
> Email_on_pre_save
> kwargs["instance"].on_pre_save()
> File "/usr/lib/python3/dist-packages/hyperkitty/models/email.py", line
> 239, in on_pre_save
> thread, _thread_created = Thread.objects.get_or_create(
> File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line
> 85, in manager_method
> return getattr(self.get_queryset(), name)(*args, **kwargs)
> File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
> 588, in get_or_create
> return self.create(**params), True
> File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
> 453, in create
> obj.save(force_insert=True, using=self.db)
> File "/usr/lib/python3/dist-packages/django/db/models/base.py", line
> 739, in save
> self.save_base(using=using, force_insert=force_insert,
> File "/usr/lib/python3/dist-packages/django/db/models/base.py", line
> 776, in save_base
> updated = self._save_table(
> File "/usr/lib/python3/dist-packages/django/db/models/base.py", line
> 881, in _save_table
> results = self._do_insert(cls._base_manager, using, fields,
> returning_fields, raw)
> File "/usr/lib/python3/dist-packages/django/db/models/base.py", line
> 919, in _do_insert
> return manager._insert(
> File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line
> 85, in manager_method
> return getattr(self.get_queryset(), name)(*args, **kwargs)
> File "/usr/lib/python3/dist-packages/django/db/models/query.py", line
> 1270, in _insert
> return query.get_compiler(using=using).execute_sql(returning_fields)
> File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py",
> line 1416, in execute_sql
> cursor.execute(sql, params)
> File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line
> 66, in execute
> return self._execute_with_wrappers(sql, params, many=False,
> executor=self._execute)
> File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line
> 75, in _execute_with_wrappers
> return executor(sql, params, many, context)
> File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line
> 84, in _execute
> return self.cursor.execute(sql, params)
> File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py",
> line 78, in execute
> raise IntegrityError(*tuple(e.args))
>
> Exception Type: IntegrityError at /hyperkitty/api/mailman/archive
> Exception Value: (1048, "Column 'starting_email_id' cannot be null")
>
> Postorius Version 1.3.8
> HyperKitty Version 1.3.7
>
> Any ideas how to fix this?
>
> Klauas
It looks like you did not run some migrations.
However, because this is packaged Mailman3, I am not sure I know how to do
that.
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
In an Internet failure case, the #1 suspect is a constant: DNS.
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
[How to ask smart questions:
http://www.catb.org/~esr/faqs/smart-questions.html]
3 months, 3 weeks
Re: Rejecting held emails
by Mark Sapiro
On 04/26/2018 08:43 AM, Darren Smith wrote:
>
> We recently had a phishing message make it through our spam filters and
> arrive at a mailing list. The good news is that since it was not from a
> list member it was in the held message queue.
>
> I was able to find the message in the var/messages directory by searching
> for a keyword in the subject. However, after rejecting the message from
> Postorius, the message doesn't seem to be removed from the var/messages
> directory.
Do you mean Reject and not Discard? Phishing/Spam should never be
"rejected" for backscatter reasons. Reject should only be used for
somehow inappropriate posts from known people.
> Is this expected behavior? I know that infosec people won't be too happy
> about a phishing email sitting around on our disk. (Infosec guys are
> notoriously paranoid. :) )
>
> Is there some sort of cleanup process that would remove these messages? If
> not, does this pose an issue where over time disk usage will creep up? This
> could be a pretty significant problem if that is the case.
It's not the behavior I would "expect", but it is the way it currently
works, and yes, all your concerns are valid. This is
<https://gitlab.com/mailman/mailman/issues/257> which I just edited to
note that in addition to not removing the pending db entry, the message
isn't removed either.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
7 years, 7 months
Re: Confusing mailman user model
by Mark Sapiro
On 7/19/20 8:10 AM, Gilles Filippini wrote:
> Hi,
>
> I'm trying to understand the Mailman user model, and I'm confused about
> the respective roles of tables 'auth_user' and 'user'.
auth_user is a Django auth table. These are the users listed in the
Django admin web UI.
user is Mailman core's users.
> Both have an 'id' column referred to as 'user_id' when used as a foreign
> key. But there seems to be no relation at all between them as they use
> different ids for the same user.
Right, they are unrelated.
> Our mailman server was migrated from mailman2 to mailman3 about 1.5 year
> ago, and there is an inconsistency I'd like to fix:
> 2 of our users have a registered Mailman account with their respective
> email addresses in table 'auth_user', but these very same addresses in
> the 'address' table are linked to another old admin account in the
> 'user' table.
Their Mailman account is the 'old' one. the auth_user account is a
Django account
>
> Then if add one of these addresses as a domain owner, it it the old
> admin address which is selected instead. How could I fix that?
That's because domain owner in a Mailman thing and involvs the Mailman
user. You could delete the Django users via the Django admin UI and then
re-register them via Postorius. That *might* work.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
5 years, 4 months
Re: Archive Policy not reflecting in Hyperkitty
by Nathan Dixon
Ok, I've figured out how to run the job manually. And here's the error I'm
getting (list name and domain has been changed to <listname> and
<domain>.com for security reasons):
From within mailman-web container:
bash-4.4# python3 manage.py runjob sync_mailman
ERROR OCCURED IN JOB: sync_mailman (APP: None)
START TRACEBACK:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line
85, in _execute
return self.cursor.execute(sql, params)
psycopg2.IntegrityError: null value in column "description" violates
not-null constraint
DETAIL: Failing row contains (4, <listname>@<domain>.com, <LISTNAME>,
null, [<LISTNAME>] , 1, 2019-03-12 10:23:20.293675+00,
<listname>.<domain>.com).
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/usr/lib/python3.6/site-packages/django_extensions/management/commands/runjob.py",
line 35, in runjob
job().execute()
File "/usr/lib/python3.6/site-packages/hyperkitty/jobs/sync_mailman.py",
line 35, in execute
sync_with_mailman()
File "/usr/lib/python3.6/site-packages/hyperkitty/lib/mailman.py", line
131, in sync_with_mailman
mlist.update_from_mailman()
File "/usr/lib/python3.6/site-packages/hyperkitty/models/mailinglist.py",
line 192, in update_from_mailman
self.save()
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line
718, in save
force_update=force_update, update_fields=update_fields)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line
748, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using,
update_fields)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line
812, in _save_table
forced_update)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line
861, in _do_update
return filtered._update(values) > 0
File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line
712, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py",
line 1383, in execute_sql
cursor = super().execute_sql(result_type)
File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py",
line 1065, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line
68, in execute
return self._execute_with_wrappers(sql, params, many=False,
executor=self._execute)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line
77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line
85, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in
__exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line
85, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "description" violates
not-null constraint
DETAIL: Failing row contains (4, <listname>@<domain.com>, <LISTNAME>,
null, [LISTNAME] , 1, 2019-03-12 10:23:20.293675+00,
<listname>.<domain>.com).
END TRACEBACK
So it looks like there's a null in one of the columns.
This list was created using the Postorius interface. I had sent a couple of
messages to the list while the archive attribute was set to public and then
I changed it to private and sent some more test mails to it.
Hopefully this will help us find a solution to this.
Thanks
On Tue, 7 May 2019 at 10:40, Nathan Dixon <nathan.dixon(a)evadon.com> wrote:
> I'm not sure if it's also related, but within the mailman-web and
> mailman-core docker containers, crontab -e only shows these entries:
> # do daily/weekly/monthly maintenance
> # min hour day month weekday command
> */15 * * * * run-parts /etc/periodic/15min
> 0 * * * * run-parts /etc/periodic/hourly
> 0 2 * * * run-parts /etc/periodic/daily
> 0 3 * * 6 run-parts /etc/periodic/weekly
> 0 5 1 * * run-parts /etc/periodic/monthly
>
> And all of those folders are empty.
>
> I'm not sure where the synchronisation task for Hyperkitty should actually
> being run from?
>
> Thank you
>
> On Tue, 7 May 2019 at 10:20, Nathan Dixon <nathan.dixon(a)evadon.com> wrote:
>
>> The change to the archiving policy aren't reflected in Hyperkitty after
>> many days for all the lists.
>>
>> The only error that I am seeing in the /web/mailmanweb.log is:
>> INFO 2019-05-02 14:16:55,062 38 hyperkitty.tasks AsyncTask task
>> "<function unlock_and_call at 0x7f5d8851f048>" with args "(<function
>> update_from_mailman at 0x7f5d88544a60>, 'task:status:%s:%s', '<list
>> address>')" and kwargs "{}" finished with errors.
>>
>> I'm not sure if that could be related? (I have changed the actual list
>> address to <list address> for security reasons)
>>
>> Thank you
>>
>> On Sat, 4 May 2019 at 16:24, Abhilash Raj <maxking(a)asynchronous.in>
>> wrote:
>>
>>> On Fri, May 3, 2019, at 8:38 AM, Nathan Dixon wrote:
>>> > So the issue I am seeing is that when I set a lists archive policy to
>>> > prviate, it isn't reflecting that in Hyperkitty.
>>> >
>>> > I am running the docker installation of Mailman, postorius and
>>> hyperkitty.
>>> > So I go into the docker instance of mailman-core and run this:
>>> >
>>> > python3 manage.py shell from hyperkitty.models.mailinglist import
>>> > MailingList MailingList.objects.all().values_list('name',
>>> 'archive_policy')
>>> >
>>> > And all my lists are showing '2', regardless of how I set the mailing
>>> list
>>> > archive policy.
>>> >
>>> > This might be a duplicate of
>>> >
>>> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/…
>>> >
>>> > Am I missing something in my install? Or is this a bug?
>>>
>>> It can take upto an hour for this change to reflect in Hyperkitty. Did
>>> you wait
>>> for that long?
>>>
>>> This happens because Hyperkitty stores it's own data, and queries Mailman
>>> Core periodically to fetch updates every hour. In future, we might be
>>> able to
>>> do this in more realtime fashion by using an event based pub/sub model.
>>> But,
>>> we are not there yet.
>>>
>>>
>>> >
>>> > Many thanks for your attention,
>>> >
>>> > --
>>> >
>>> > *Nathan Dixon* MEng
>>> > Senior Software Architect
>>> > m: +44 (0)7402 690311
>>> >
>>> > t: +44 (0)20 81231252
>>> > e: nathan.dixon(a)evadon.com
>>> > w: www.evadon.com
>>> >
>>> > This e-mail and any attachments are confidential and may be protected
>>> by
>>> > legal, professional or other privilege. If you are not the intended
>>> > recipient you should not store it, copy it, re-transmit it, use it or
>>> > disclose its contents, but should return it to the sender immediately
>>> and
>>> > delete your copy from your system. The views expressed are those of the
>>> > sender and not necessarily those of Evadon. Please note that whilst we
>>> scan
>>> > all e-mails for viruses we cannot guarantee that any e-mail is
>>> virus-free.
>>> > Please be advised that we expressly reserve the right to monitor email
>>> > content for the purposes of ensuring compliance with legal
>>> requirements and
>>> > company policies and your sending to, or receiving from, us of any
>>> email
>>> > constitutes your agreement to these terms.
>>> >
>>> >
>>> > Evadon Dynamics Limited, registered in England No. 09494097. Registered
>>> > Office: Electric Works, Sheffield Digital Campus, 3 Concourse Way,
>>> Sheffield,
>>> > S1 2BJ
>>> > _______________________________________________
>>> > Mailman-users mailing list -- mailman-users(a)mailman3.org
>>> > To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>>> > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>>> >
>>>
>>> --
>>> thanks,
>>> Abhilash Raj (maxking)
>>> _______________________________________________
>>> Mailman-users mailing list -- mailman-users(a)mailman3.org
>>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>>>
>>
>>
>> --
>>
>> *Nathan Dixon* MEng
>> Senior Software Architect
>> m: +44 (0)7402 690311
>>
>> t: +44 (0)20 81231252
>> e: nathan.dixon(a)evadon.com
>> w: www.evadon.com
>>
>> This e-mail and any attachments are confidential and may be protected by
>> legal, professional or other privilege. If you are not the intended
>> recipient you should not store it, copy it, re-transmit it, use it or
>> disclose its contents, but should return it to the sender immediately and
>> delete your copy from your system. The views expressed are those of the
>> sender and not necessarily those of Evadon. Please note that whilst we scan
>> all e-mails for viruses we cannot guarantee that any e-mail is virus-free.
>> Please be advised that we expressly reserve the right to monitor email
>> content for the purposes of ensuring compliance with legal requirements and
>> company policies and your sending to, or receiving from, us of any email
>> constitutes your agreement to these terms.
>>
>>
>> Evadon Dynamics Limited, registered in England No. 09494097. Registered
>> Office: Electric Works, Sheffield Digital Campus, 3 Concourse Way, Sheffield,
>> S1 2BJ
>>
>
>
> --
>
> *Nathan Dixon* MEng
> Senior Software Architect
> m: +44 (0)7402 690311
>
> t: +44 (0)20 81231252
> e: nathan.dixon(a)evadon.com
> w: www.evadon.com
>
> This e-mail and any attachments are confidential and may be protected by
> legal, professional or other privilege. If you are not the intended
> recipient you should not store it, copy it, re-transmit it, use it or
> disclose its contents, but should return it to the sender immediately and
> delete your copy from your system. The views expressed are those of the
> sender and not necessarily those of Evadon. Please note that whilst we scan
> all e-mails for viruses we cannot guarantee that any e-mail is virus-free.
> Please be advised that we expressly reserve the right to monitor email
> content for the purposes of ensuring compliance with legal requirements and
> company policies and your sending to, or receiving from, us of any email
> constitutes your agreement to these terms.
>
>
> Evadon Dynamics Limited, registered in England No. 09494097. Registered
> Office: Electric Works, Sheffield Digital Campus, 3 Concourse Way, Sheffield,
> S1 2BJ
>
--
*Nathan Dixon* MEng
Senior Software Architect
m: +44 (0)7402 690311
t: +44 (0)20 81231252
e: nathan.dixon(a)evadon.com
w: www.evadon.com
This e-mail and any attachments are confidential and may be protected by
legal, professional or other privilege. If you are not the intended
recipient you should not store it, copy it, re-transmit it, use it or
disclose its contents, but should return it to the sender immediately and
delete your copy from your system. The views expressed are those of the
sender and not necessarily those of Evadon. Please note that whilst we scan
all e-mails for viruses we cannot guarantee that any e-mail is virus-free.
Please be advised that we expressly reserve the right to monitor email
content for the purposes of ensuring compliance with legal requirements and
company policies and your sending to, or receiving from, us of any email
constitutes your agreement to these terms.
Evadon Dynamics Limited, registered in England No. 09494097. Registered
Office: Electric Works, Sheffield Digital Campus, 3 Concourse Way, Sheffield,
S1 2BJ
6 years, 7 months
Re: mailman-web migrate fails with sqlite3.IntegrityError: UNIQUE constraint failed: account_emailaddress.email
by Mark Sapiro
On 3/3/25 07:52, Stefan Bauer via Mailman-users wrote:
>
> trying to update mailman3 to lastest version, we fail during
>
> mailman-web migrate with:
>
> Operations to perform:
> Apply all migrations: account, admin, auth, contenttypes, django_mailman3,
> django_q, hyperkitty, postorius, sessions, sites, socialaccount
> Running migrations:
> Applying account.0006_emailaddress_lower...Traceback (most recent call
> last):
> File
> "/opt/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> line 89, in _execute
> return self.cursor.execute(sql, params)
> File
> "/opt/mailman/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py",
> line 328, in execute
> return super().execute(query, params)
> sqlite3.IntegrityError: UNIQUE constraint failed: account_emailaddress.email
This issue is due to changes in Django allauth to prevent creating
multiple addresses differing only in case. The problem is if you have
such multiple addresses in your account_emailaddress table, this
migration fails. There is a script at
https://www.msapiro.net/scripts/UC_fix that will fix this. You may need
to adjust the shebang and replace psycopg2 with sqlite3.
See https://github.com/pennersr/django-allauth/issues/3019 perhaps
starting at
https://github.com/pennersr/django-allauth/issues/3019#issuecomment-1748028…
> We found a workaround to fake the migration with
>
> migrate --fake
You should fix the issue with the DB and run the migrations.
> Could this be some duplicate users with same mail-address?
This is not related to Mailman users. It is Django users.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
9 months
Re: Performance of member listing [was: How to remove a nonmember...]
by Abhilash Raj
On Sat, Apr 18, 2020, at 8:25 AM, skenny(a)scss.tcd.ie wrote:
> Well, it is quite possible that something is seriously wrong with my
> configuration.
>
> The database backend is Postgres - not something I configured. It is
> specified in my docker-compose file, which in turn is based on the
> compose files at https://github.com/maxking/docker-mailman.
>
> (At one point I thought maybe the virtual machine I'm running on was
> under-resourced. I've since doubled memory and the number of CPUs but
> that hasn't really improved things.)
>
> I'm wondering where to go next - try tweaking the Postgres config,
> switch to SQLlite ....
Switching to SQLite will definitely degrade performance, unless Postgres database is being weird. I haven't observed any issues reported in the past for the performance.
I am a little confused which part are we talking here about being too slow. You mentioned two things:
- The python script you wrote using MailmanClient
- Listing non-members/members
- Delete a non-member via Postorius.
It would be nice to establish which specific view is being slow so I can dig more into it. Can you paste the URL and action which takes 2.5minutes as you mentioned?
>
> Thanks for giving time to helping me with this, much appreciated.
>
> Stephen
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>
--
thanks,
Abhilash Raj (maxking)
5 years, 7 months