Re: Newbie hyperkitty question
by Mark Sapiro
On 1/11/22 8:27 AM, Stephen Daniel wrote:
> Mailman users --
>
> I am setting up a new mailman3 installation, and this is my first
> experience with the suite. All of the software is up and running, but I
> cannot tell if hyperkitty is actually running.
In Postorius, in the list's `info` view can you click the Archives
button and see archived messages? That's HyperKitty.
If the messages aren't there with the hyperkitty archiving option
enabled, check `mailman.log` for why.
Also do messages from the list have an Archived-At: header?
> When I look at my test mailing list, two archive options are available,
> hyperkitty and "prototype", and both are enabled. I can see that my email
> is being archived, but I think "prototype" is what is doing the archiving.
> I see that a test message will end up in
> /opt/mailman/mm/var/archives/prototype/<LISTNAME>@<LISTDOMAIN.org>/new/164....
> and nowhere else. This suggests that "prototype" is archiving, but
> hyperkitty is not.
No this says the prototype archiver is working. It says nothing about
Hyperkitty.
> So, this opens up a raft of questions on hyperkitty:
David Newman has answered some of this, but ...
> - How do I tell if hyperkitty is actually running?
It doesn't actually run. it is a Django web UI and a web API that serves
things on request.
> - Does hyperkitty have a web-UI for admins or users? How would I access
> it?
See above.
> - I can not find any documentation on hyperkitty, other than
> installation instructions and change logs. Does documentation exist?
No, not really.
> - How do I set policies on how long archives are kept?
> - Assuming I want to back up my archives, what do I need to back up?
Messages are kept forever unless you do something to delete them.
HyperKitty's messages are stored in the database configured in the
DATABASES setting in your Django settings. How you backup the database
depends on the database manager. You already found the prototype
archiver's messages.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
4 years, 3 months
Re: How to store additional information about lists?
by Andreas Barth
* Stephen J. Turnbull (turnbull.stephen.fw(a)u.tsukuba.ac.jp) [210523 02:41]:
> Andreas Barth writes:
>
> > Currently it is "list public" vs "subscribers only". I need something
> > inbetween, so that some (i.e. most) people have access to the
> > restricted audience lists but not just everybody.
>
> That much is clear. The problem is that "access to list" can mean a
> lot of things: can subscribe, can post, can admin, can view archives,
> can view list in listinfo, can view subscriber list, and there are
> probably others. What permissions are needed for what general classes
> of users?
public: anyone can subscribe and read the archive. Think about an
announcement mailing list of an organisation.
restricted: for employees (or members - or whatever) only. So they
could subscribe themself. The archive and posting may be restricted to
subscribers or to anyone in that group - both would be working for me.
(How to know who is allowed to subscribe is another question - having
an internal mailing list which contains all the members just for this
purpose would be ok for me.)
private: hand-selected people, think about e.g. board or steering
committees internal discussions. Of course, the archive (if there is
any) should be restricted to the actual members. Posting to these
lists is currently not restricted, so no derivation needed from the
current set in postorius.
> > Just as an idea, how about always one "custom attributes" field. So
> > that from a database pov this field is always there.
>
> We could do that, but then we'd have to support it. I'm thinking of
> situations like admin A populates the field, then leaves the
> organization, and we add (some of) the supported features, and admin B
> wants to migrate that. I don't see any good coming of that for us.
> As Mark says, a separate table seems like the way to go for custom
> attributes.
Well, that problem will always exist. I don't think you would make it
harder by having one "custom attributes" field.
I'm going to try the plugin route Mark suggested.
Andi
4 years, 11 months
Re: [MM3-users off list] Re: Welcome message template errors
by Mark
>> So, I'm thinking (in broad brush strokes) that if I'm stuck with this
>> "http glitch" I could write a script to:
>>
>> 1. extract the text of the custom message from the database,
>> 2. write that message text to a file
>> 3. update the templates.uri entries to replace the
>> http://localhost... with file:///path/to/file
>> 4. cron the script to run every few minutes.
>>
>> It screams of utter clunk I know...
> but it would work.
>
> Note that this may not be as big an issue as it seems. It doesn't
> affect all templates, for example list:member:digest:footer,
> list:member:digest:header, list:member:digest:masthead,
> list:member:regular:footer, list:member:regular:header and
> list:member:generic:footer are only used by the `out` runner when
> sending messages, so these aren't affected.
Right. Affected are all the "list:user:notice:***" (eg. goodbye, hold,
reject,...)
>
> I'm still trying to determine what the issue is. To this end can you
> tell me what Python version your Mailman uses and what is the output
> from `pip freeze` with your venv active?
$ mailman info
GNU Mailman 3.3.9 (Tom Sawyer)
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
$ pip freeze
aiosmtpd==1.4.4.post2
alembic==1.13.1
arrow==1.3.0
asgiref==3.7.2
atpublic==4.0
attrs==23.2.0
authheaders==0.16.2
authres==1.2.0
blessed==1.20.0
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cmarkgfm==2024.1.14
cryptography==42.0.1
defusedxml==0.7.1
Django==4.2.11
django-allauth==0.60.1
django-appconf==1.0.6
django-compressor==4.4
django-extensions==3.2.3
django-gravatar2==1.4.4
django-haystack==3.2.1
django-mailman3==1.3.12
django-picklefield==3.1
django-q==1.3.9
djangorestframework==3.14.0
dkimpy==1.1.5
dnspython==2.5.0
docutils==0.20.1
falcon==3.1.3
filelock==3.13.1
flufl.bounce==4.0
flufl.i18n==5.0.2
flufl.lock==8.0.2
greenlet==3.0.3
gunicorn==21.2.0
HyperKitty==1.3.9
idna==3.6
lazr.config==3.0
lazr.delegates==2.1.0
mailman==3.3.9
mailman-hyperkitty==1.2.1
mailman-web==0.0.9
mailmanclient==3.3.5
Mako==1.3.0
MarkupSafe==2.1.4
mistune==3.0.2
networkx==3.2.1
nh3==0.2.15
oauthlib==3.2.2
packaging==23.2
passlib==1.7.4
postorius==1.3.10
psutil==5.9.8
psycopg2-binary==2.9.9
publicsuffix2==2.20191221
pycparser==2.21
Pygments==2.17.2
PyJWT==2.8.0
python-dateutil==2.8.2
python3-openid==3.2.0
pytz==2023.3.post1
rcssmin==1.1.1
readme-renderer==42.0
redis==3.5.3
requests==2.31.0
requests-oauthlib==1.3.1
rjsmin==1.2.1
robot-detection==0.4
six==1.16.0
SQLAlchemy==2.0.25
sqlparse==0.4.4
types-python-dateutil==2.8.19.20240106
typing_extensions==4.9.0
urllib3==2.1.0
wcwidth==0.2.13
Whoosh==2.7.4
xapian @ file:///opt/mailman/xapian-1.4.24-cp311-cp311-linux_aarch64.whl
xapian-haystack==3.1.0
zope.component==6.0
zope.configuration==5.0
zope.event==5.0
zope.hookable==6.0
zope.i18nmessageid==6.1.0
zope.interface==6.1
zope.schema==7.0.1
2 years
Re: converting bulk accept_these_nonmembers in migration from mailman 2 to 3
by Lucio Chiappetti
Dear Mark, thanks for your interest.
On Fri, 11 Feb 2022, Mark Sapiro wrote:
> This may actually be a bug. I'll have to think about that. I've filed
> https://gitlab.com/mailman/mailman/-/issues/978 on this.
> This is https://gitlab.com/mailman/mailman/-/issues/794 (still open)
Given the time needed to solve the issues (I will anyhow notify the site
administrators, hopting they would update mailman when/if resolved), this
means we have to find a workaround.
> And those non-members all have there moderation action set to Defer
> which means their posts will be accepted but the additional checks such
> as too big, etc. will still be applied.
not sure what you mean by "defer". The non-member options I see are called
List default -- follow the list's default member action.
Hold -- This holds the message for approval by the list moderators.
Reject -- this automatically rejects the message by sending a bounce
notice to the post's author. The text of the bounce notice can be
configured by you.
Discard -- this simply discards the message, with no notice sent to the
post's author.
Accept -- accepts any postings without any further checks.
Default Processing -- run additional checks and accept the message.
> If they are nonmembers, you can accept the post and set moderation
> action in one operation in Postorius, but the regexps in
> hold_these_nonmembers will still take precedence.
OK, I've found where to do it, it requires to select the held message and
scroll to the bottom then set Moderation from a menu.
But if hold_these_nonmembers takes priority this will be uselesws in the
current setting.
> I'm not sure what you are looking at here. Imported nonmembers should
> all have moderation action set according to which *_these_nonmembers
> they came from. Imported members should have their moderation action set
> to Defer if they were unmoderated and if they were moderated, it should
> be set based on the 2.1 list's member_moderation_action.
I go to the non-member list, select non-member options and at the bottom
see an item Administration options Moderation. The possible values are
those I pasted above.
The actual value for most of the imported non-members is "Default
processing" while it is "List default" for thoase automatically added
afresh, It is "Discard" for a number of imported non-members with
spam-looking addresses (I guess they were in some other *_not_members part
of the standard 2.1 antispam ... I can't recall me doing something on
those)
Side question: is there a way to operate IN BULK on non-members ? Like it
is for members (I exported them to CSV).
>> - if so, how can we do it automatically for all 189 entries ?
>> - or move back the 189 addresses to accept_these_nonmembers ?
> Yes, move them back.
I am not so concerned of these 189 (or better of the 189 minus the discard
ones which should stay) ... anyhow moving them back will be uncomfortable
if there is not a bulk operation tool.
I am more concerned of the new cases automatically added.
Our member list includes 1043 addresses. Of these less than 20 use an
external address (say gmail), so they post "from home" and will pass.
194 addresses are of the form name.surname(a)inaf.it (these are likely to be
recently hired staff, who have and use onlyu the organization-wide
address).
All the rest (so still more than 800, can't really ask all of them to
re-register (*)) are of the form username(a)institute.inaf.it (and it won't
be easy also to bulk-delete and bulk-resubscribe because of the
inhomogeneous mapping of username to name.surname.
Of these about 800 some are lurkers and are no problem. Some are frequent
posters and remember to post from username(a)institute.inaf.it and are no
problem. Other may post from an alternate address (maybe even the old
address exists as a receiving-only alias), and in this case with
the current arrangement they will bo on hold every time.
I guess we should wait for the solution of issue 794
(*) by the way do you confirm that if one is subscribed to a list with an
address (be it bulk subscription, import or new fast subscription), this
person CANNOT change their settings (or address) unless one does a "sign
in" to a (Postorius) account ?
> It is because all the legacy *_these_nonmember actions are applied
> before nonmember moderation checks, so if an address matches a regex in
> hold_these_nonmembers, the post will be held regardless of the
> nonmember's moderation action.
> Addresses still work, and accept_these_nonmembers takes precedence over
> hold_these_nonmembers just as in MM 2.1.
This was not my impression when I ran a thorough test of 12 cases. Except
the obvious case of the subscriber posting from the subscription address,
all messages went on hold.
Whatever the non-member moderation would be (even if set to reject ! ...
but that makes sense if *_these_nonmembers prevail on non-member options),
and even if the address was set explicitly in accept_these_nonmembers
(should I delete it from the non-member list ?)
What is the actual order (or flow chart) of choices ?
*_these_nonmembers in the order listed in the page (within each in the
order of occurrence of regexp's) then non-member options ?
If I have Name.Surname(a)inaf.it in accept_these_nonmembers, then
hold_these_nonmembers set to
^.+@.+\.it
^.+@.+\.tng\.iac\.es
what do you foresee ?
And for different regexps, will processing terminate at the first
non-matching one, will they be ANDed or ORed, or what ?
I am reluctant to accept inconditionally ^.+\..+(a)inaf\.it (they may be
falsified, only the check on their presence in the other list (issue 794)
I guess I should remove the ^.+@.+\.it regexp AND AT THE SAME TIME change
the Default action to take when a non-member posts to the list to Hold
instead of Discard.
This way we'll have to check spam (instead of having it auto-discarded),
but if people with alternate addresses post, once they are entered in the
Non-member list they can be authorized to pass for the future ...
.. would it work ?
--
Lucio Chiappetti - INAF/IASF - via Corti 12 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
4 years, 2 months
Re: Mailman, etc. upgrade woes and persistent bugs
by Abhilash Raj
I am trying to distill some information from this discussion to be
used for the implementation in Postorius. I'll maybe reiterate a bit
of what Steve was saying just to confirm I am on the same page.
On Tue, Feb 16, 2021, at 7:23 AM, Stephen J. Turnbull wrote:
> No answer required, I think I understand the situation pretty well
> now. But if I'm missing something, I would very much appreciate
> criticism.
>
> Andreas Barth writes:
>
> > Well, currently these are exclusive-states, I'd rather see them as
> > or-able-states.
>
> Right, I understand that. I don't understand *why*.
>
> There are real costs to making them or-able. There are constraints:
> 'enabled' shouldn't (mustn't?) be or-able with any 'disabled-by-foo'
> state. Such invariants are finicky to code and tricky to maintain,
> compared to a simple "mutually exclusive".
>
> I guess you want to allow the or of any subset of disabled states, but
> presumably then they need to individually resettable in order for the
> admin to clear their disable without overriding a subscriber's
> preference or a bounce record. That set of controls is more complex,
> and requires additional mental effort on the part of anyone using
> them. If there are restrictions on which states can be or-ed, that's
> more coding and maintenance cost.
>
> Then there's the question whether the subscriber can override the admin's
> setting. If they can, why does the admin have this power in the first
> place?
>
> > > > - "disabled by member" (could be set by an admin as well of course)
> > >
> > > Currently this is determined by whether the logged in user is the
> > > subscriber or not. It's an interesting idea to allow it to be set by
> > > an admin. Then we could have the semantics that the only transition
> > > allowed from this state would be to enabled, so that admins would know
> > > the user had intentionally disabled and know not to enable without
> > > permission.
> >
> > Exactly.
>
> But if the admin should respect the DBS (disabled by subscriber)
> setting, DBS|DBA (disabled by admin) is logically equivalent to DBS.
> Are there use cases where the admin does not want to repect the
> subscriber's choice, and enables it anyway? What are they?
>
> > Or while importing from another server, and there are people who
> > have receiving mails disabled, this is the state that could be used
> > for it.
>
> Why wouldn't they get the exact state from the original server? Or do
> you mean from a non-Mailman server?
>
> > I expect that state to be the most-often used when setting a state
> > manually.
> >
> > For "who set that", I would consider having some kind of history as
> > useful (and displaying "last set by user / $person" to the admin but
> > not the user).
>
> Why not to the subscriber? As I explain below, the subscriber at
> least needs to be able to distinguish between DBA and DBB (disabled by
> bounce).
>
> Keeping history of who is additional complexity, consumes a minor
> amount of resources, and is insufficient to the purpose. Assuming
> that there are reasons for admins to disable/enable other than as a
> courtesy to a subscriber, that reason is presumably important. Not only
> will the subscriber not know, but one admin may forget, and there may be
> multiple admins.[1]
>
> I still don't see any reason for an admin to disable delivery other
> than as a courtesy to the subscriber or to the bounce processor.
>
> > > > - "disabled for administrative reasons" (only by admin but user
> > > > should be able to see it)
> > >
> > > I don't see why a user would need to see the difference. Unless
> > > you're suggesting that the user would not be able to reenable without
> > > asking the admin? What is the use case for this?
> >
> > I'm suggesting exactly that, yes. Why? Well, worked with user for too
> > long. ;)
> >
> > E.g. consider that seeing archives works only for subscribers. And the
> > subscriber in question complains about too many mails but still
> > re-enables getting mails. This is basically "an admin has blocked you
> > from more mails but not unsubscribed you". Might be an corner case,
> > but still. Might also be too unimportant.
>
> This is possible. If it occurs less than once in a million years,
> it's "too unimportant" to even think about.[2] Have you actually done
> this, or know anybody who has?
>
> Is it really different from "at request of subscriber"? Practically,
> the only difference I can see is if such a subscriber should be
> prevented from overriding DBA. Are you suggesting that?
>
> > > > - "bounced" (user and admin could reset it but not set it)
> > >
> > > Reenable but not set "bounced" is the current situation.
> >
> > Except if someone sets the state to "disabled" and then undo this, the
> > state is automatically cleared.
>
> This is possible, but it's not clear what harm would be done. In most
> cases, bounces are intermittent and clear themselves up, in which case
> this is the right thing. But if the subscriber bounces again, they'll
> get disabled again, only cost is a slight waste of resources.
>
> Here's how I would assess the issues. In the descriptions below, I am
> assuming that each state is the only "active" state.
>
> 1. DBS is probably a "vacation" or "post-only" setting. For that
> reason it should be re-enabled only by the subscriber (or by an
> admin at the request of the subscriber).
> 2. DBB is a "warning" and a minor convenience to the email system
> (slightly reducing traffic to mailboxes that are expected to fail,
> and perhaps avoiding some reputational damage to the list).
> Re-enabling much of the time causes no damage, as bounce disables
> are typically due to things like out of disk space, and are likely
> fixed by the time the subscription is reenabled. Most of the rest
> will be zombie mailboxes, which will rarely be reenabled, will
> quickly be redisabled by bounce, and at worst cause mild
> inconvenience. Finally, events like a Mailman upgrade and the
> April 2014 DMARC fiasco will be pretty evident, and the damage
> there has already been done, reenabling actually fixes it
> (although it may not be effective if the systemic problem hasn't
> been resolved).
> 3. DBA is interesting mostly to *subscribers*, who need to
> distinguish DBA (which the subscriber can reenable freely) from
> DBB (which warns the subscriber to get in touch with their
> postmaster). It's comforting to subscribers who may be sure they
> never disabled their subscription; otherwise, it's semantically
> identical to DBS as far as I can see.
>
> Am I missing something?
>
> It seems to me that given 1-3, assuming a 4-state design:
>
> 0. It's never unreasonable to re-enable, except DBS.
> 1. In the DBA state, changing to either DBS or DBB causes no
> problems. (Changing to DBB is unlikely, since no mail is being
> sent. However, it's possible, due to the race between Postorius
> and mail in transit, and due to spam and other spoofing of the
> list domain.)
> 2. In either of the other two states, we should never change to DBA.
> 3. In either of the other two states, if the other event occurs, you
> would want the state to be DBS|DBB. I contend that if we
> substitute DBS, little harm is done. If the subscriber reenables, it's
> on them to fix the bounce problem if it reoccurs in any case.
> Since the subscriber has deliberately set DBS, the admin should not
> reenable without permission of the subscriber (I think we're mostly in
> agreement on that).
>
> I think this 4-state design is much easier to design, validate, and
> maintain than the 16-state design you propose. If you're really
> worried about DBS|DBB, then we could have a 5-state design where
> enabling from DBS|DBB generates a warning that there may still be a
> bounce problem on the subscription. Instead, it could leave the
> subscription in state DBB, but that seems to me to be obnoxious, since
> there's little likelihood of harm from reenabling, and in many cases
> the bounce problem will have resolved itself in the meantime.
>
So it seems like we have some sort of agreement that we need all the
roles to be able to see all the 4 states. Roles here are basically Admin(
i.e. list owner/moderator/site admin) and Subscriber.
There is no access control to transition "from" any state but each value
of non-enabled states is sort of a role's way to disable the delivery. Not
all states are settable by all roles.
1. A subscriber can only transition from any to "enabled" or "by_user"
state but can see if delivery was disabled "by_admin" or "by_bounces"
as separate state in the Options page.
2. An admin can transition to "enabled", "by_user" and "by_moderator".
They should use "by_user" when acting on behalf of the user and other
when there are any administrative reason to disable delivery, whatever
that may be. I expect "by_admin" to be used sparingly, and I don't see
any use cases for it.
3. Mailman itself will use the "by_bounces" to disable delivery due to
bounces.
Is that an accurate description?
--
thanks,
Abhilash Raj (maxking)
5 years, 2 months
Re: Mass operations on large list - Internal Server Error
by Stephen J. Turnbull
Reilly, Liam writes:
> Here is an example error from the log for unsubscribe all
> operation.
Which log is this? I would guess it's the mailmanweb.log.
> [2025-12-15 15:41:02 +0000] [1022464] [ERROR] Error handling request /mailman3/lists/uclucu-members.ucl.ac.uk/unsubscribe_all
> Traceback (most recent call last):
> File "/data/mailman/venv/lib64/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
> self.handle_request(listener, req, client, addr)
This is the front-end gunicorn WSGI server, handling a request for
Postorius.
> File "/data/mailman/venv/lib64/python3.12/site-packages/mailmanclient/restobjects/mailinglist.py", line 519, in unsubscribe
> response, json = self._connection.call(path, data, method='DELETE')
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is mailman client translating to the REST API.
> File "/usr/lib64/python3.12/socket.py", line 720, in readinto
> return self._sock.recv_into(b)
> ^^^^^^^^^^^^^^^^^^^^^^^
This is the Python socket library actually talking to the REST API of
Mailman core. An unspecified error occurs in the C code. It is not
handled.
> File "/data/mailman/venv/lib64/python3.12/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
> sys.exit(1)
> SystemExit: 1
This is the "Something's wrong, hope you have a clue 'cause I don't"
last line of defense error handler.
My guess is that it's a timeout on a socket.
I think you should also see log messages and a traceback in the
mailman.log.
The timeout for the mailmanweb gunicorn is configured in the gunicorn
conf file, usually in /etc/mailman3. This is relatively short (I
think my installation is default at 120 seconds).
The timeout for the REST API gunicorn is in
site-packages/mailman/config/gunicorn.cfg. This is long (360 seconds)
so if my conjecture about timeout is correct it's the mailmanweb
gunicorn that's timing out. If you need to change this one, you may
be able to simply put "timeout: 360" in the [webservice] section of
your mailman.cfg file, but if that doesn't work you can copy
gunicorn.cfg to /etc/mailman3/rest-wsgi.cfg, change the timeout in it,
and add "configuration: /etc/mailman3/rest-wsgi.cfg" to the
[webservice] section of mailman.cfg.
--
GNU Mailman consultant (installation, migration, customization)
Sirius Open Source https://www.siriusopensource.com/
Software systems consulting in Europe, North America, and Japan
4 months, 1 week
Templates branch for Mailman 3.1
by Barry Warsaw
Hello developers!
There are three big features still in progress for Mailman 3.1, the templates
branch, MySQL support, and unsubscription workflow. There is already a merge
request for MySQL support and I'm hoping to merge that soon. Abhilash is also
working on the unsubscription workflow feature but there's no MR for that yet.
The templates feature is very nearly done, so I think now is a good time to
ask for testing, feedback, and review. This is a generalization of the
welcome_message_uri feature in MM3.0 where you could specify e.g. the welcome
message as a URL to download and cache when necessary. This is important
because Core doesn't know whether there is even a web ui (e.g. Postorius)
involved, or where things like clickable links to unsubscribe would be. Sites
also want to customize footers to include information like codes of conduct,
etc.
Now (or when this branch lands), you will be able to specify a URL to retrieve
any template that Mailman uses internally, whether they are confirmation
messages, bounce probes, welcome and goodbye messages, etc. These can be
templates for administrators, users, and members.
Because I'm hoping to get some reviews, I'll point you at the documentation
for more details:
https://gitlab.com/warsaw/mailman/blob/templates/src/mailman/rest/docs/temp…
The branch did end up being much bigger and tentacled than I expected, but
it's all hanging together pretty well now, and all the tests pass. There's
still a small number of to-dos, but some of them will get punted, and the
others won't have a major effect on the overall use and visibility of the
feature.
https://gitlab.com/warsaw/mailman/blob/templates/TODO.rst
Note also that if you interact with the REST API v3.0, nothing changes
externally, but the old IMailingList attributes are mapped onto the new
templates mechanism. API v3.1 is where you see all the actual new feature
end-points exposed. Of course, any MM2 list converted to MM3.1 will use the
new templates stuff too.
Feedback very much welcome, either here or on the MR:
https://gitlab.com/mailman/mailman/merge_requests/170
I'd like to land this some time in the next week or so. Then I'll work on the
other two big branches and start to release some betas.
Cheers,
-Barry
9 years, 9 months
Re: Problems after update to mailman 3.3.7
by Danil Smirnov
Hi Guillermo,
We've faced the same issue and already reported it:
https://gitlab.com/mailman/mailman/-/issues/1044
I would not recommend anyone using MySQL/MariaDB database engine upgrade to
the latest version yet.
With my best regards,
Danil Smirnov
Mailman3.com
On Tue, Dec 6, 2022 at 4:32 PM Guillermo Hernandez (Oldno7) via
Mailman-users <mailman-users(a)mailman3.org> wrote:
> I've getting the most strange behaviour after update to last releases.
> In one of my mailman3 servers, the messages doesnt reach for some lists
> (the five last ones)
>
> the update have not showed any error but in the var/data for the lmtp
> and vmap postfix files mailman does not create aliases for that last 5
> lists. Any message addressed to them is returned with a "no such user"
> error.
>
> If I create a new list the system or restart mailman runners it shows
> this error:
>
> Generating MTA alias maps
> /usr/local/lib/python3.9/site-packages/pymysql/connections.py:799:
> UserWarning: Previous unbuffered result was left incomplete
> warnings.warn("Previous unbuffered result was left incomplete")
>
> I did a CHECK TABLE on mailinglists table but all seems ok.
>
> I tried reinstalling PyMySQL, but has no effect. (well, I tried
> reinstalling all)
>
> It seems to me that something is preventing creation of that aliases, an
> then the that last five lists are unreachable.
>
> I have no clue of what to do now and I'm driving crazy.
>
> Some direction to look for will be much apreciated.
>
> Thanks in advance.
>
> P.S.: I can access via postorius/hyperkitty to the archives of the lists
> that are unreacheable.
>
>
> --
>
> ___________________________________________
> Mailman's content filtering has removed the
> following MIME parts from this message.
>
> Content-Type: image/png
> Name: firma-GHP-emails.png
>
> Replaced multipart/alternative part with first alternative.
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at:
> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>
> This message sent to danil(a)smirnov.la
>
3 years, 4 months
Re: Issues with Mail delivery after Mailman 3 installation
by Chihurumnaya Ibiam
On Thu, Feb 5, 2026 at 4:35 AM Stephen J. Turnbull <steve(a)turnbull.jp>
wrote:
> Chihurumnaya Ibiam via Mailman-users writes:
> > On Wed, Feb 4, 2026 at 9:06 PM Stephen J. Turnbull <steve(a)turnbull.jp>
> > wrote:
> >
> > > Chihurumnaya Ibiam via Mailman-users writes:
>
> > > > systemctl shows the service is still active,
> > > The fact that there is a systemd service implies that dovecot
> > > will be running by default unless it is disabled.
> > There's no service file for dovecot, I didn't create one.
>
> As far as I know, if systemd knows about a service, there's a control
> file somewhere. systemd will create a unit from an rc file in
> /etc/rcN.d if there is one. (I was perfectly happy with SysV init,
> but I guess systemd makes hyperscalars happy.) I don't know if
> "systemctl disable" works on those.
>
I don't plan on shutting down the host anytime soon, so hopefully, I
remember to the next time it restarts.
I started out with Fedora the year before it adopted systemd, so I never
really knew about SysV init until today.
> > I configured mailbox transport because that was how I was able to get
> lmtp
> > to run on port 24,
>
> Something like that would be necessary for Dovecot, which I believe
> can be configured to automatically create a mailbox for any address.
> It is not necessary for Mailman, which tells Postfix how to deliver to
> Mailman lists in the postfix_lmtp table. You can find that in
> /opt/mailman/mm/var/data, if you followed the virtualenv installation
> to the letter.
>
Yes, postfix_lmtp table is set in transport_maps, virtual_mailbox_maps,
relay_domains, local_recipient_maps.
>
> It's problematic for Mailman because that means that *all* local mail
> will be delivered to Mailman, which will reject it if it's not
> list-related.
>
I'm totally fine with this because that's why we setup the host, although
one thing I noticed in the logs was that other servers
trying to connect to the host was rejected, they're trying to send logs to
a mailing list, which should work just fine but then it seems
they need to connect to this host.
connect to weblate.sugarlabs.org[192.184.220.214]:25: Connection refused
165968 Feb 5 04:48:46 lists postfix/smtp[1941010]: connect to
weblate.sugarlabs.org[2001:5a8:601:f::214]:25: Connection refused
165969 Feb 5 04:48:46 lists postfix/smtp[1941010]: B785D1A897F: to=<
root(a)weblate.sugarlabs.org>, relay=none, delay=17323,
delays=17323/0.02/0.33/0, dsn=4.4.1, status=deferred (connect to
weblate.sugarlabs.org[2001:5a8:601:f::214]:25: Connection refused)
I tried to fix this with check_client_access
hash:/etc/postfix/client_access, which contains a wild card for the main
domain and subdomains,
which seems to have no effect.
> I'm not sure this is why the confirmation mails are getting rejected,
> but at a guess, for some classes of mail Postfix will first try the
> whole address, and if that fails to route, it will remove the
> extension (starting with "+"), and try again. Perhaps it does NOT do
> that for the mailbox_transport.
>
> > there wasn't any service before that and the documentation
> > <
> https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.h…
> >
> > for setting up a mail server mentions postfix delivering
> > by default over port 24.
>
> What it says is this:
>
> Postfix will deliver via LMTP over port 24 by default, however if
> you are not running Mailman as root, you’ll need to change this to
> a higher port number, as shown above.
> Are you running Mailman as root? If so, stop it, change lmtp_port
> back to 8024 in mailman.cfg, and stop and start Mailman to regenerate
> the postfix_lmtp.db. There's no need for running as root unless you
> have an MTA that is hard-coded to think that 24 is the one and only
> lmtp port handed down from Mount Olympus. Postfix knows better.
>
No, I'm not running mailman as root, it's run as mailman.
Seeing as that's the case, would I still need to change lmtp_port?
>
> Furthermore, you should NOT run mailman as root if there's any
> alternative. Mailman occasionally crashes for various reasons (I've
> never seen it take Python down, but then, if there were an exploit, I
> wouldn't), and is not securely authenticated (I mean, it's reasonably
> secure, but it is just a password, and once you get in to Mailman
> core, you're the superuser for Mailman) and if there *is* an exploit
> and Mailman is running as root, you've got system root, too.
>
> Don't run Mailman as root. Please.
>
I haven't, I was never going to.
>
> > Disabling this means there's no lmtp to be used by postfix, except I'm
> > missing something.
>
> Yes, you're missing something. Postfix is always able to speak LMTP,
> and is very flexible about when to do so. If you followed the virtualenv
> installation to the letter, you have
>
> transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp
>
> in main.cf (perhaps there's other maps there), and in that file you
> have lines that end in "lmtp:[127.0.0.1]:24", which tell Postfix
> "connect to IP address 127.0.0.1, port 24, and speak LMTP to that
> connection." If you change lmtp_port to 8024 and restart Mailman as
> suggested, all those lines will change to "lmtp:[127.0.0.1]:8024" and
> Postfix will be happy to deliver to Mailman that way without being root.
>
After your comment about Postfix being an LMTP server, I understood this
better.
>
> > I set the port number for lmtp to be 24, and that's what I used, I'm not
> > sure if Mailman sets up lmtp itself,
> > if it does then I'll have to remove the config for that.
> >
> > I've stopped dovecot after our conversation so it's no longer running,
> the
> > only way the port is open is through the
> > config I set in postfix for mailbox_transport.
>
> That should get it working, except that with mailbox_transport set
> there's a good chance that Mailman functions that use "+" extensions
> will mess up as described above. Those functions are confirmations
> and I think VERP probes.
>
>
It did get it working, I can see mail being sent in the logs.
I sent a ping to the systems list just to get delivery confirmation and I'm
yet to get a response.
> > There's just one active user, which is me, and I also have root access.
> > I setup authentication for postfix because I'm used to setting it up
> that
> > way, this was a force of habit.
> > I assumed postfix would run just fine if I set postfix up the way I
> usually
> > set it up.
>
> It might. But "Internet Mail" is another name for Dante's Third Circle
> of Hell. (You must have committed a horrible felony to be sentenced
> to administer an Internet mail host. :-D That's just a joke, it
> sounds to me like your "usual setup" is quite solid and secure.)
> Anyway, any deviation from the setup described in virtualenv.html is a
> chance for everything to go wrong. "What I usually do" is just not
> deep enough knowledge if you're working with a new mail application
> with complex routing needs.
>
😅😅 Funniest thing I've read today.
I agree with your last sentence, needed to be reminded of that.
>
> > > What does "lsof -i @127.0.0.1 -i '@[::1]' | grep 24" output?
>
> > python3 1592922 mailman 27u IPv4 177685024 0t0 TCP
> localhost:8001 (LISTEN)
> > python3 1592949 mailman 27u IPv4 177685024 0t0 TCP
> localhost:8001 (LISTEN)
> > python3 1592950 mailman 27u IPv4 177685024 0t0 TCP
> localhost:8001 (LISTEN)
>
> The above are Mailman (actually, a gunicorn application) listening for
> connections to its REST API. These are HTTP listeners used by
> Postorius and HyperKitty (I guess those aren't running?), not for email.
>
Postorius and HyperKitty are running.
> > postgres 3568035 postgres 5u IPv6 160224851 0t0 TCP
> localhost:postgresql (LISTEN)
> > postgres 3568035 postgres 6u IPv4 160224852 0t0 TCP
> localhost:postgresql (LISTEN)
>
> These are PostgreSQL listening for database connections.
>
> > python3 1592924 mailman 22u IPv6 177685051 0t0 TCP
> localhost:48638->localhost:postgresql (ESTABLISHED)
>
> This is Mailman talking to Postfix.
> > postgres 1592954 postgres 9u IPv6 177686124 0t0 TCP
> localhost:postgresql->localhost:48498 (ESTABLISHED)
> > postgres 1592912 postgres 8u IPv6 160224857 0t0 UDP
> localhost:36787->localhost:36787
>
> + 21 more copies of the above, differing only in the PID. Don't know
> what's going on with those, not a PostgreSQL wizard. I probably
> should have specified TCP in the lsof command, UDP applications are
> either trivial or they are black magic.
>
> It appears that *nothing* is listening on or connected to port 24.
> I'm not sure how that works. I'm guessing that when mail was getting
> to Mailman, Mailman had opened port 24 and Dovecot was cut out of the
> loop (maybe check logs for Dovecot complaining it couldn't bind to
> port 24?)
>
The logs don't show Dovecot complaining about binding to port 24, the only
thing.
This is one that's consistent with Dovecot in the logs;
157322 Feb 4 12:27:02 lists postfix/cleanup[1593312]: AC7381A8978:
message-id=<20260204202702.AC7381A8978(a)lists.sugarlabs.org>
157323 Feb 4 12:27:02 lists postfix/bounce[1587396]: A26F71A8977: sender
non-delivery notification: AC7381A8978
157324 Feb 4 12:27:02 lists postfix/qmgr[1593314]: AC7381A8978: from=<>,
size=3129, nrcpt=1 (queue active)
157325 Feb 4 12:27:02 lists postfix/local[1593315]: AC7381A8978: passing <
mailman(a)lists.sugarlabs.org> to transport=lmtp
157326 Feb 4 12:27:02 lists dovecot: lmtp(1600482): Connect from 127.0.0.1
157327 Feb 4 12:27:02 lists postfix/qmgr[1593314]: A26F71A8977: removed
157328 Feb 4 12:27:02 lists postfix/lmtp[1593316]: AC7381A8978: to=<
mailman(a)lists.sugarlabs.org>, relay=localhost[127.0.0.1]:24, delay=0.04,
delays=0.02/0/0/0.01, dsn=5.1.1, status=bounced (host localhost[127.0.0.1]
said: 550 5.1.1 <mailman(a)lists.sugarlabs.org> User doesn't exist:
mailman(a)lists.sugarlabs.org (in reply to RCPT TO command))
157329 Feb 4 12:27:02 lists dovecot: lmtp(1600482): Disconnect from
127.0.0.1: Logged out (state=READY)
Which we've established that mailman doesn't know how to deliver to local
users.
>
> > > > [mta] lmtp_host: *.*.*.*
> > >
> > > That should be 127.0.0.1. Mailman should not be listening on network
> > > interfaces other than the loopback (localhost).
> >
> > Yes, it is.
>
> OK
>
> > > > [mta] lmtp_port: 24
> > >
> > > This can't work if Dovecot is configured to use the same port, as is
> > > Dovecot's default.
>
> > I've stopped the service.
>
> OK
>
> > > > [mta] outgoing: mailman.mta.deliver.deliver
> > > > [mta] smtp_host: 127.0.0.1
> > > > [mta] smtp_port: 25
> > >
> > > Those are normal.
> > >
> > > > [mta] smtp_pass: **************
> > > > [mta] smtp_user: mailman
> > >
> > > This is going to cause Mailman to try to authenticate to Postfix.
> > > That's OK if it's what you want and Postfix / saslauthd has been
> > > properly configured to accept Mailman's credentials. Otherwise,
> > > they must be left empty to disable authentication.
>
> > Okay.
>
> > > > The Message-ID being the way it is doesn't seem like a
> > > > configuration issue, if it is then how do I look into it?
> > >
> > > Probably you don't have the full hostname in Postfix's myorigin or
> > > myhostname parameter.
> >
> > I do have the full hostname there, I also have it in /etc/mailname.
>
> I don't understand it then. Perhaps the mail composition client is
> providing the Message-ID.
>
Could be, doesn't seem to be an issue at the moment so I won't worry about
it.
>
> > > If not, I would guess that's a spammer who is somehow spoofing
> > > the PTR record.
>
> > The log entry before that warned of the same, they're definitely a
> spammer.
> > Relay access is denied though, so no mails are going through.
>
> OK.
>
> > No, I mean smtps.
>
> Same problem except the port that Mailman is *not* using is 465.
>
Yes, earlier when I set it up I had set the port for Mailman to use as 465.
>
> > > I don't think permit_auth_destination does what you think it does.
> > > Here's the Postfix doc:
> > >
> > > permit_auth_destination
> > > Permit the request when one of the following is true:
> > >
> > > - Postfix is a mail forwarder: the resolved RCPT TO domain
> > > matches $relay_domains or a subdomain thereof, and the address
> > > contains no sender-specified routing (user@elsewhere@domain),
> > > - Postfix is the final destination: the resolved RCPT TO domain
> > > matches $mydestination, $inet_interfaces, $proxy_interfaces,
> > > $virtual_alias_domains, or $virtual_mailbox_domains, and the
> > > address contains no sender-specified routing
> > > (user@elsewhere@domain).
> > >
> >
> > I did read the documentation before adding it, I definitely
> misunderstood
> > what it meant, and after looking at the old
> > documentation, it makes sense not to add it.
> >
> > Could you help me understand what the above means?
>
> I gotta run now, I'll come back to it later. You're probably asleep,
> anyway. :-)
>
> --
> GNU Mailman consultant (installation, migration, customization)
> Sirius Open Source https://www.siriusopensource.com/
> Software systems consulting in Europe, North America, and Japan
--
Ibiam Chihurumnaya
ibiamchihurumnaya(a)gmail.com
2 months, 3 weeks
Re: Mailman, etc. upgrade woes and persistent bugs
by Stephen J. Turnbull
Andreas Barth writes:
> * Stephen J. Turnbull (turnbull.stephen.fw(a)u.tsukuba.ac.jp) [210217 09:54]:
> > Right. So what you mean is "ok, we have group permissions, and maybe
> > someday there will be a case for ACLs down to arbitrary subsets of the
> > user population," but you don't have a concrete use case for arbitrary
> > ACLs at the moment.
> I definitly have a usecase for that, but: currently other priorities :)
> (I had to do an ad-hoc takeover of lots of infrastructure in my spare
> time within an volunteer organisation including mailing lists, so
> getting wishlist itmes done is not my top priority at the moment.)
>
> Just from the top of my head, for example I want to differentiate
> between private, member and public lists. So if a member logs in, they
> see more lists than a non-member. This is currently not possible, not
> too big a deal (all member lists are currently configured as private)
> but of course would be nice. Also archiv access should be aligned with
> these permissions.
This would not be worth doing with generic ACLs from Postorius, unless
the list had (many) fewer than 100 users. What I'd end up doing (and I
bet you would too) is write a 15-line mailmancli script that extracted
membership information and then created the ACLs. But you'd still
have to modify the Django templates, I suspect, and perhaps database
schema.
Essentially the same 15 lines could be put into Mailman itself without
ACLs, and checking ACL membership is not going to be much, if at all,
faster than checking list membership.
> Of course the appropriate split between private changes and public
> changes is also something which might need some discussions.
Not sure what you mean by private and public here, but if you mean
which changes we should maintain and which changes you should maintain
locally, I haven't seen anything in this thread that we wouldn't want
to offer all Mailman 3 users. From Abhilash's and Mark's posts, I
believe they feel that way too. It's just a question of whether a
very general implementation is needed/worthwhile.
Steve
5 years, 2 months