Re: [MM3-users off list] Re: Welcome message template errors
by Mark
>> So, I'm thinking (in broad brush strokes) that if I'm stuck with this
>> "http glitch" I could write a script to:
>>
>> 1. extract the text of the custom message from the database,
>> 2. write that message text to a file
>> 3. update the templates.uri entries to replace the
>> http://localhost... with file:///path/to/file
>> 4. cron the script to run every few minutes.
>>
>> It screams of utter clunk I know...
> but it would work.
>
> Note that this may not be as big an issue as it seems. It doesn't
> affect all templates, for example list:member:digest:footer,
> list:member:digest:header, list:member:digest:masthead,
> list:member:regular:footer, list:member:regular:header and
> list:member:generic:footer are only used by the `out` runner when
> sending messages, so these aren't affected.
Right. Affected are all the "list:user:notice:***" (eg. goodbye, hold,
reject,...)
>
> I'm still trying to determine what the issue is. To this end can you
> tell me what Python version your Mailman uses and what is the output
> from `pip freeze` with your venv active?
$ mailman info
GNU Mailman 3.3.9 (Tom Sawyer)
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
$ pip freeze
aiosmtpd==1.4.4.post2
alembic==1.13.1
arrow==1.3.0
asgiref==3.7.2
atpublic==4.0
attrs==23.2.0
authheaders==0.16.2
authres==1.2.0
blessed==1.20.0
certifi==2023.11.17
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
cmarkgfm==2024.1.14
cryptography==42.0.1
defusedxml==0.7.1
Django==4.2.11
django-allauth==0.60.1
django-appconf==1.0.6
django-compressor==4.4
django-extensions==3.2.3
django-gravatar2==1.4.4
django-haystack==3.2.1
django-mailman3==1.3.12
django-picklefield==3.1
django-q==1.3.9
djangorestframework==3.14.0
dkimpy==1.1.5
dnspython==2.5.0
docutils==0.20.1
falcon==3.1.3
filelock==3.13.1
flufl.bounce==4.0
flufl.i18n==5.0.2
flufl.lock==8.0.2
greenlet==3.0.3
gunicorn==21.2.0
HyperKitty==1.3.9
idna==3.6
lazr.config==3.0
lazr.delegates==2.1.0
mailman==3.3.9
mailman-hyperkitty==1.2.1
mailman-web==0.0.9
mailmanclient==3.3.5
Mako==1.3.0
MarkupSafe==2.1.4
mistune==3.0.2
networkx==3.2.1
nh3==0.2.15
oauthlib==3.2.2
packaging==23.2
passlib==1.7.4
postorius==1.3.10
psutil==5.9.8
psycopg2-binary==2.9.9
publicsuffix2==2.20191221
pycparser==2.21
Pygments==2.17.2
PyJWT==2.8.0
python-dateutil==2.8.2
python3-openid==3.2.0
pytz==2023.3.post1
rcssmin==1.1.1
readme-renderer==42.0
redis==3.5.3
requests==2.31.0
requests-oauthlib==1.3.1
rjsmin==1.2.1
robot-detection==0.4
six==1.16.0
SQLAlchemy==2.0.25
sqlparse==0.4.4
types-python-dateutil==2.8.19.20240106
typing_extensions==4.9.0
urllib3==2.1.0
wcwidth==0.2.13
Whoosh==2.7.4
xapian @ file:///opt/mailman/xapian-1.4.24-cp311-cp311-linux_aarch64.whl
xapian-haystack==3.1.0
zope.component==6.0
zope.configuration==5.0
zope.event==5.0
zope.hookable==6.0
zope.i18nmessageid==6.1.0
zope.interface==6.1
zope.schema==7.0.1
1 year, 10 months
Re: converting bulk accept_these_nonmembers in migration from mailman 2 to 3
by Lucio Chiappetti
Dear Mark, thanks for your interest.
On Fri, 11 Feb 2022, Mark Sapiro wrote:
> This may actually be a bug. I'll have to think about that. I've filed
> https://gitlab.com/mailman/mailman/-/issues/978 on this.
> This is https://gitlab.com/mailman/mailman/-/issues/794 (still open)
Given the time needed to solve the issues (I will anyhow notify the site
administrators, hopting they would update mailman when/if resolved), this
means we have to find a workaround.
> And those non-members all have there moderation action set to Defer
> which means their posts will be accepted but the additional checks such
> as too big, etc. will still be applied.
not sure what you mean by "defer". The non-member options I see are called
List default -- follow the list's default member action.
Hold -- This holds the message for approval by the list moderators.
Reject -- this automatically rejects the message by sending a bounce
notice to the post's author. The text of the bounce notice can be
configured by you.
Discard -- this simply discards the message, with no notice sent to the
post's author.
Accept -- accepts any postings without any further checks.
Default Processing -- run additional checks and accept the message.
> If they are nonmembers, you can accept the post and set moderation
> action in one operation in Postorius, but the regexps in
> hold_these_nonmembers will still take precedence.
OK, I've found where to do it, it requires to select the held message and
scroll to the bottom then set Moderation from a menu.
But if hold_these_nonmembers takes priority this will be uselesws in the
current setting.
> I'm not sure what you are looking at here. Imported nonmembers should
> all have moderation action set according to which *_these_nonmembers
> they came from. Imported members should have their moderation action set
> to Defer if they were unmoderated and if they were moderated, it should
> be set based on the 2.1 list's member_moderation_action.
I go to the non-member list, select non-member options and at the bottom
see an item Administration options Moderation. The possible values are
those I pasted above.
The actual value for most of the imported non-members is "Default
processing" while it is "List default" for thoase automatically added
afresh, It is "Discard" for a number of imported non-members with
spam-looking addresses (I guess they were in some other *_not_members part
of the standard 2.1 antispam ... I can't recall me doing something on
those)
Side question: is there a way to operate IN BULK on non-members ? Like it
is for members (I exported them to CSV).
>> - if so, how can we do it automatically for all 189 entries ?
>> - or move back the 189 addresses to accept_these_nonmembers ?
> Yes, move them back.
I am not so concerned of these 189 (or better of the 189 minus the discard
ones which should stay) ... anyhow moving them back will be uncomfortable
if there is not a bulk operation tool.
I am more concerned of the new cases automatically added.
Our member list includes 1043 addresses. Of these less than 20 use an
external address (say gmail), so they post "from home" and will pass.
194 addresses are of the form name.surname(a)inaf.it (these are likely to be
recently hired staff, who have and use onlyu the organization-wide
address).
All the rest (so still more than 800, can't really ask all of them to
re-register (*)) are of the form username(a)institute.inaf.it (and it won't
be easy also to bulk-delete and bulk-resubscribe because of the
inhomogeneous mapping of username to name.surname.
Of these about 800 some are lurkers and are no problem. Some are frequent
posters and remember to post from username(a)institute.inaf.it and are no
problem. Other may post from an alternate address (maybe even the old
address exists as a receiving-only alias), and in this case with
the current arrangement they will bo on hold every time.
I guess we should wait for the solution of issue 794
(*) by the way do you confirm that if one is subscribed to a list with an
address (be it bulk subscription, import or new fast subscription), this
person CANNOT change their settings (or address) unless one does a "sign
in" to a (Postorius) account ?
> It is because all the legacy *_these_nonmember actions are applied
> before nonmember moderation checks, so if an address matches a regex in
> hold_these_nonmembers, the post will be held regardless of the
> nonmember's moderation action.
> Addresses still work, and accept_these_nonmembers takes precedence over
> hold_these_nonmembers just as in MM 2.1.
This was not my impression when I ran a thorough test of 12 cases. Except
the obvious case of the subscriber posting from the subscription address,
all messages went on hold.
Whatever the non-member moderation would be (even if set to reject ! ...
but that makes sense if *_these_nonmembers prevail on non-member options),
and even if the address was set explicitly in accept_these_nonmembers
(should I delete it from the non-member list ?)
What is the actual order (or flow chart) of choices ?
*_these_nonmembers in the order listed in the page (within each in the
order of occurrence of regexp's) then non-member options ?
If I have Name.Surname(a)inaf.it in accept_these_nonmembers, then
hold_these_nonmembers set to
^.+@.+\.it
^.+@.+\.tng\.iac\.es
what do you foresee ?
And for different regexps, will processing terminate at the first
non-matching one, will they be ANDed or ORed, or what ?
I am reluctant to accept inconditionally ^.+\..+(a)inaf\.it (they may be
falsified, only the check on their presence in the other list (issue 794)
I guess I should remove the ^.+@.+\.it regexp AND AT THE SAME TIME change
the Default action to take when a non-member posts to the list to Hold
instead of Discard.
This way we'll have to check spam (instead of having it auto-discarded),
but if people with alternate addresses post, once they are entered in the
Non-member list they can be authorized to pass for the future ...
.. would it work ?
--
Lucio Chiappetti - INAF/IASF - via Corti 12 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
4 years, 1 month
Re: Mailman, etc. upgrade woes and persistent bugs
by Abhilash Raj
I am trying to distill some information from this discussion to be
used for the implementation in Postorius. I'll maybe reiterate a bit
of what Steve was saying just to confirm I am on the same page.
On Tue, Feb 16, 2021, at 7:23 AM, Stephen J. Turnbull wrote:
> No answer required, I think I understand the situation pretty well
> now. But if I'm missing something, I would very much appreciate
> criticism.
>
> Andreas Barth writes:
>
> > Well, currently these are exclusive-states, I'd rather see them as
> > or-able-states.
>
> Right, I understand that. I don't understand *why*.
>
> There are real costs to making them or-able. There are constraints:
> 'enabled' shouldn't (mustn't?) be or-able with any 'disabled-by-foo'
> state. Such invariants are finicky to code and tricky to maintain,
> compared to a simple "mutually exclusive".
>
> I guess you want to allow the or of any subset of disabled states, but
> presumably then they need to individually resettable in order for the
> admin to clear their disable without overriding a subscriber's
> preference or a bounce record. That set of controls is more complex,
> and requires additional mental effort on the part of anyone using
> them. If there are restrictions on which states can be or-ed, that's
> more coding and maintenance cost.
>
> Then there's the question whether the subscriber can override the admin's
> setting. If they can, why does the admin have this power in the first
> place?
>
> > > > - "disabled by member" (could be set by an admin as well of course)
> > >
> > > Currently this is determined by whether the logged in user is the
> > > subscriber or not. It's an interesting idea to allow it to be set by
> > > an admin. Then we could have the semantics that the only transition
> > > allowed from this state would be to enabled, so that admins would know
> > > the user had intentionally disabled and know not to enable without
> > > permission.
> >
> > Exactly.
>
> But if the admin should respect the DBS (disabled by subscriber)
> setting, DBS|DBA (disabled by admin) is logically equivalent to DBS.
> Are there use cases where the admin does not want to repect the
> subscriber's choice, and enables it anyway? What are they?
>
> > Or while importing from another server, and there are people who
> > have receiving mails disabled, this is the state that could be used
> > for it.
>
> Why wouldn't they get the exact state from the original server? Or do
> you mean from a non-Mailman server?
>
> > I expect that state to be the most-often used when setting a state
> > manually.
> >
> > For "who set that", I would consider having some kind of history as
> > useful (and displaying "last set by user / $person" to the admin but
> > not the user).
>
> Why not to the subscriber? As I explain below, the subscriber at
> least needs to be able to distinguish between DBA and DBB (disabled by
> bounce).
>
> Keeping history of who is additional complexity, consumes a minor
> amount of resources, and is insufficient to the purpose. Assuming
> that there are reasons for admins to disable/enable other than as a
> courtesy to a subscriber, that reason is presumably important. Not only
> will the subscriber not know, but one admin may forget, and there may be
> multiple admins.[1]
>
> I still don't see any reason for an admin to disable delivery other
> than as a courtesy to the subscriber or to the bounce processor.
>
> > > > - "disabled for administrative reasons" (only by admin but user
> > > > should be able to see it)
> > >
> > > I don't see why a user would need to see the difference. Unless
> > > you're suggesting that the user would not be able to reenable without
> > > asking the admin? What is the use case for this?
> >
> > I'm suggesting exactly that, yes. Why? Well, worked with user for too
> > long. ;)
> >
> > E.g. consider that seeing archives works only for subscribers. And the
> > subscriber in question complains about too many mails but still
> > re-enables getting mails. This is basically "an admin has blocked you
> > from more mails but not unsubscribed you". Might be an corner case,
> > but still. Might also be too unimportant.
>
> This is possible. If it occurs less than once in a million years,
> it's "too unimportant" to even think about.[2] Have you actually done
> this, or know anybody who has?
>
> Is it really different from "at request of subscriber"? Practically,
> the only difference I can see is if such a subscriber should be
> prevented from overriding DBA. Are you suggesting that?
>
> > > > - "bounced" (user and admin could reset it but not set it)
> > >
> > > Reenable but not set "bounced" is the current situation.
> >
> > Except if someone sets the state to "disabled" and then undo this, the
> > state is automatically cleared.
>
> This is possible, but it's not clear what harm would be done. In most
> cases, bounces are intermittent and clear themselves up, in which case
> this is the right thing. But if the subscriber bounces again, they'll
> get disabled again, only cost is a slight waste of resources.
>
> Here's how I would assess the issues. In the descriptions below, I am
> assuming that each state is the only "active" state.
>
> 1. DBS is probably a "vacation" or "post-only" setting. For that
> reason it should be re-enabled only by the subscriber (or by an
> admin at the request of the subscriber).
> 2. DBB is a "warning" and a minor convenience to the email system
> (slightly reducing traffic to mailboxes that are expected to fail,
> and perhaps avoiding some reputational damage to the list).
> Re-enabling much of the time causes no damage, as bounce disables
> are typically due to things like out of disk space, and are likely
> fixed by the time the subscription is reenabled. Most of the rest
> will be zombie mailboxes, which will rarely be reenabled, will
> quickly be redisabled by bounce, and at worst cause mild
> inconvenience. Finally, events like a Mailman upgrade and the
> April 2014 DMARC fiasco will be pretty evident, and the damage
> there has already been done, reenabling actually fixes it
> (although it may not be effective if the systemic problem hasn't
> been resolved).
> 3. DBA is interesting mostly to *subscribers*, who need to
> distinguish DBA (which the subscriber can reenable freely) from
> DBB (which warns the subscriber to get in touch with their
> postmaster). It's comforting to subscribers who may be sure they
> never disabled their subscription; otherwise, it's semantically
> identical to DBS as far as I can see.
>
> Am I missing something?
>
> It seems to me that given 1-3, assuming a 4-state design:
>
> 0. It's never unreasonable to re-enable, except DBS.
> 1. In the DBA state, changing to either DBS or DBB causes no
> problems. (Changing to DBB is unlikely, since no mail is being
> sent. However, it's possible, due to the race between Postorius
> and mail in transit, and due to spam and other spoofing of the
> list domain.)
> 2. In either of the other two states, we should never change to DBA.
> 3. In either of the other two states, if the other event occurs, you
> would want the state to be DBS|DBB. I contend that if we
> substitute DBS, little harm is done. If the subscriber reenables, it's
> on them to fix the bounce problem if it reoccurs in any case.
> Since the subscriber has deliberately set DBS, the admin should not
> reenable without permission of the subscriber (I think we're mostly in
> agreement on that).
>
> I think this 4-state design is much easier to design, validate, and
> maintain than the 16-state design you propose. If you're really
> worried about DBS|DBB, then we could have a 5-state design where
> enabling from DBS|DBB generates a warning that there may still be a
> bounce problem on the subscription. Instead, it could leave the
> subscription in state DBB, but that seems to me to be obnoxious, since
> there's little likelihood of harm from reenabling, and in many cases
> the bounce problem will have resolved itself in the meantime.
>
So it seems like we have some sort of agreement that we need all the
roles to be able to see all the 4 states. Roles here are basically Admin(
i.e. list owner/moderator/site admin) and Subscriber.
There is no access control to transition "from" any state but each value
of non-enabled states is sort of a role's way to disable the delivery. Not
all states are settable by all roles.
1. A subscriber can only transition from any to "enabled" or "by_user"
state but can see if delivery was disabled "by_admin" or "by_bounces"
as separate state in the Options page.
2. An admin can transition to "enabled", "by_user" and "by_moderator".
They should use "by_user" when acting on behalf of the user and other
when there are any administrative reason to disable delivery, whatever
that may be. I expect "by_admin" to be used sparingly, and I don't see
any use cases for it.
3. Mailman itself will use the "by_bounces" to disable delivery due to
bounces.
Is that an accurate description?
--
thanks,
Abhilash Raj (maxking)
5 years
Re: Mass operations on large list - Internal Server Error
by Stephen J. Turnbull
Reilly, Liam writes:
> Here is an example error from the log for unsubscribe all
> operation.
Which log is this? I would guess it's the mailmanweb.log.
> [2025-12-15 15:41:02 +0000] [1022464] [ERROR] Error handling request /mailman3/lists/uclucu-members.ucl.ac.uk/unsubscribe_all
> Traceback (most recent call last):
> File "/data/mailman/venv/lib64/python3.12/site-packages/gunicorn/workers/sync.py", line 134, in handle
> self.handle_request(listener, req, client, addr)
This is the front-end gunicorn WSGI server, handling a request for
Postorius.
> File "/data/mailman/venv/lib64/python3.12/site-packages/mailmanclient/restobjects/mailinglist.py", line 519, in unsubscribe
> response, json = self._connection.call(path, data, method='DELETE')
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This is mailman client translating to the REST API.
> File "/usr/lib64/python3.12/socket.py", line 720, in readinto
> return self._sock.recv_into(b)
> ^^^^^^^^^^^^^^^^^^^^^^^
This is the Python socket library actually talking to the REST API of
Mailman core. An unspecified error occurs in the C code. It is not
handled.
> File "/data/mailman/venv/lib64/python3.12/site-packages/gunicorn/workers/base.py", line 204, in handle_abort
> sys.exit(1)
> SystemExit: 1
This is the "Something's wrong, hope you have a clue 'cause I don't"
last line of defense error handler.
My guess is that it's a timeout on a socket.
I think you should also see log messages and a traceback in the
mailman.log.
The timeout for the mailmanweb gunicorn is configured in the gunicorn
conf file, usually in /etc/mailman3. This is relatively short (I
think my installation is default at 120 seconds).
The timeout for the REST API gunicorn is in
site-packages/mailman/config/gunicorn.cfg. This is long (360 seconds)
so if my conjecture about timeout is correct it's the mailmanweb
gunicorn that's timing out. If you need to change this one, you may
be able to simply put "timeout: 360" in the [webservice] section of
your mailman.cfg file, but if that doesn't work you can copy
gunicorn.cfg to /etc/mailman3/rest-wsgi.cfg, change the timeout in it,
and add "configuration: /etc/mailman3/rest-wsgi.cfg" to the
[webservice] section of mailman.cfg.
--
GNU Mailman consultant (installation, migration, customization)
Sirius Open Source https://www.siriusopensource.com/
Software systems consulting in Europe, North America, and Japan
2 months, 3 weeks
Templates branch for Mailman 3.1
by Barry Warsaw
Hello developers!
There are three big features still in progress for Mailman 3.1, the templates
branch, MySQL support, and unsubscription workflow. There is already a merge
request for MySQL support and I'm hoping to merge that soon. Abhilash is also
working on the unsubscription workflow feature but there's no MR for that yet.
The templates feature is very nearly done, so I think now is a good time to
ask for testing, feedback, and review. This is a generalization of the
welcome_message_uri feature in MM3.0 where you could specify e.g. the welcome
message as a URL to download and cache when necessary. This is important
because Core doesn't know whether there is even a web ui (e.g. Postorius)
involved, or where things like clickable links to unsubscribe would be. Sites
also want to customize footers to include information like codes of conduct,
etc.
Now (or when this branch lands), you will be able to specify a URL to retrieve
any template that Mailman uses internally, whether they are confirmation
messages, bounce probes, welcome and goodbye messages, etc. These can be
templates for administrators, users, and members.
Because I'm hoping to get some reviews, I'll point you at the documentation
for more details:
https://gitlab.com/warsaw/mailman/blob/templates/src/mailman/rest/docs/temp…
The branch did end up being much bigger and tentacled than I expected, but
it's all hanging together pretty well now, and all the tests pass. There's
still a small number of to-dos, but some of them will get punted, and the
others won't have a major effect on the overall use and visibility of the
feature.
https://gitlab.com/warsaw/mailman/blob/templates/TODO.rst
Note also that if you interact with the REST API v3.0, nothing changes
externally, but the old IMailingList attributes are mapped onto the new
templates mechanism. API v3.1 is where you see all the actual new feature
end-points exposed. Of course, any MM2 list converted to MM3.1 will use the
new templates stuff too.
Feedback very much welcome, either here or on the MR:
https://gitlab.com/mailman/mailman/merge_requests/170
I'd like to land this some time in the next week or so. Then I'll work on the
other two big branches and start to release some betas.
Cheers,
-Barry
9 years, 8 months
Re: Problems after update to mailman 3.3.7
by Danil Smirnov
Hi Guillermo,
We've faced the same issue and already reported it:
https://gitlab.com/mailman/mailman/-/issues/1044
I would not recommend anyone using MySQL/MariaDB database engine upgrade to
the latest version yet.
With my best regards,
Danil Smirnov
Mailman3.com
On Tue, Dec 6, 2022 at 4:32 PM Guillermo Hernandez (Oldno7) via
Mailman-users <mailman-users(a)mailman3.org> wrote:
> I've getting the most strange behaviour after update to last releases.
> In one of my mailman3 servers, the messages doesnt reach for some lists
> (the five last ones)
>
> the update have not showed any error but in the var/data for the lmtp
> and vmap postfix files mailman does not create aliases for that last 5
> lists. Any message addressed to them is returned with a "no such user"
> error.
>
> If I create a new list the system or restart mailman runners it shows
> this error:
>
> Generating MTA alias maps
> /usr/local/lib/python3.9/site-packages/pymysql/connections.py:799:
> UserWarning: Previous unbuffered result was left incomplete
> warnings.warn("Previous unbuffered result was left incomplete")
>
> I did a CHECK TABLE on mailinglists table but all seems ok.
>
> I tried reinstalling PyMySQL, but has no effect. (well, I tried
> reinstalling all)
>
> It seems to me that something is preventing creation of that aliases, an
> then the that last five lists are unreachable.
>
> I have no clue of what to do now and I'm driving crazy.
>
> Some direction to look for will be much apreciated.
>
> Thanks in advance.
>
> P.S.: I can access via postorius/hyperkitty to the archives of the lists
> that are unreacheable.
>
>
> --
>
> ___________________________________________
> Mailman's content filtering has removed the
> following MIME parts from this message.
>
> Content-Type: image/png
> Name: firma-GHP-emails.png
>
> Replaced multipart/alternative part with first alternative.
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at:
> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>
> This message sent to danil(a)smirnov.la
>
3 years, 3 months
Re: Issues with Mail delivery after Mailman 3 installation
by Chihurumnaya Ibiam
On Thu, Feb 5, 2026 at 4:35 AM Stephen J. Turnbull <steve(a)turnbull.jp>
wrote:
> Chihurumnaya Ibiam via Mailman-users writes:
> > On Wed, Feb 4, 2026 at 9:06 PM Stephen J. Turnbull <steve(a)turnbull.jp>
> > wrote:
> >
> > > Chihurumnaya Ibiam via Mailman-users writes:
>
> > > > systemctl shows the service is still active,
> > > The fact that there is a systemd service implies that dovecot
> > > will be running by default unless it is disabled.
> > There's no service file for dovecot, I didn't create one.
>
> As far as I know, if systemd knows about a service, there's a control
> file somewhere. systemd will create a unit from an rc file in
> /etc/rcN.d if there is one. (I was perfectly happy with SysV init,
> but I guess systemd makes hyperscalars happy.) I don't know if
> "systemctl disable" works on those.
>
I don't plan on shutting down the host anytime soon, so hopefully, I
remember to the next time it restarts.
I started out with Fedora the year before it adopted systemd, so I never
really knew about SysV init until today.
> > I configured mailbox transport because that was how I was able to get
> lmtp
> > to run on port 24,
>
> Something like that would be necessary for Dovecot, which I believe
> can be configured to automatically create a mailbox for any address.
> It is not necessary for Mailman, which tells Postfix how to deliver to
> Mailman lists in the postfix_lmtp table. You can find that in
> /opt/mailman/mm/var/data, if you followed the virtualenv installation
> to the letter.
>
Yes, postfix_lmtp table is set in transport_maps, virtual_mailbox_maps,
relay_domains, local_recipient_maps.
>
> It's problematic for Mailman because that means that *all* local mail
> will be delivered to Mailman, which will reject it if it's not
> list-related.
>
I'm totally fine with this because that's why we setup the host, although
one thing I noticed in the logs was that other servers
trying to connect to the host was rejected, they're trying to send logs to
a mailing list, which should work just fine but then it seems
they need to connect to this host.
connect to weblate.sugarlabs.org[192.184.220.214]:25: Connection refused
165968 Feb 5 04:48:46 lists postfix/smtp[1941010]: connect to
weblate.sugarlabs.org[2001:5a8:601:f::214]:25: Connection refused
165969 Feb 5 04:48:46 lists postfix/smtp[1941010]: B785D1A897F: to=<
root(a)weblate.sugarlabs.org>, relay=none, delay=17323,
delays=17323/0.02/0.33/0, dsn=4.4.1, status=deferred (connect to
weblate.sugarlabs.org[2001:5a8:601:f::214]:25: Connection refused)
I tried to fix this with check_client_access
hash:/etc/postfix/client_access, which contains a wild card for the main
domain and subdomains,
which seems to have no effect.
> I'm not sure this is why the confirmation mails are getting rejected,
> but at a guess, for some classes of mail Postfix will first try the
> whole address, and if that fails to route, it will remove the
> extension (starting with "+"), and try again. Perhaps it does NOT do
> that for the mailbox_transport.
>
> > there wasn't any service before that and the documentation
> > <
> https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.h…
> >
> > for setting up a mail server mentions postfix delivering
> > by default over port 24.
>
> What it says is this:
>
> Postfix will deliver via LMTP over port 24 by default, however if
> you are not running Mailman as root, you’ll need to change this to
> a higher port number, as shown above.
> Are you running Mailman as root? If so, stop it, change lmtp_port
> back to 8024 in mailman.cfg, and stop and start Mailman to regenerate
> the postfix_lmtp.db. There's no need for running as root unless you
> have an MTA that is hard-coded to think that 24 is the one and only
> lmtp port handed down from Mount Olympus. Postfix knows better.
>
No, I'm not running mailman as root, it's run as mailman.
Seeing as that's the case, would I still need to change lmtp_port?
>
> Furthermore, you should NOT run mailman as root if there's any
> alternative. Mailman occasionally crashes for various reasons (I've
> never seen it take Python down, but then, if there were an exploit, I
> wouldn't), and is not securely authenticated (I mean, it's reasonably
> secure, but it is just a password, and once you get in to Mailman
> core, you're the superuser for Mailman) and if there *is* an exploit
> and Mailman is running as root, you've got system root, too.
>
> Don't run Mailman as root. Please.
>
I haven't, I was never going to.
>
> > Disabling this means there's no lmtp to be used by postfix, except I'm
> > missing something.
>
> Yes, you're missing something. Postfix is always able to speak LMTP,
> and is very flexible about when to do so. If you followed the virtualenv
> installation to the letter, you have
>
> transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp
>
> in main.cf (perhaps there's other maps there), and in that file you
> have lines that end in "lmtp:[127.0.0.1]:24", which tell Postfix
> "connect to IP address 127.0.0.1, port 24, and speak LMTP to that
> connection." If you change lmtp_port to 8024 and restart Mailman as
> suggested, all those lines will change to "lmtp:[127.0.0.1]:8024" and
> Postfix will be happy to deliver to Mailman that way without being root.
>
After your comment about Postfix being an LMTP server, I understood this
better.
>
> > I set the port number for lmtp to be 24, and that's what I used, I'm not
> > sure if Mailman sets up lmtp itself,
> > if it does then I'll have to remove the config for that.
> >
> > I've stopped dovecot after our conversation so it's no longer running,
> the
> > only way the port is open is through the
> > config I set in postfix for mailbox_transport.
>
> That should get it working, except that with mailbox_transport set
> there's a good chance that Mailman functions that use "+" extensions
> will mess up as described above. Those functions are confirmations
> and I think VERP probes.
>
>
It did get it working, I can see mail being sent in the logs.
I sent a ping to the systems list just to get delivery confirmation and I'm
yet to get a response.
> > There's just one active user, which is me, and I also have root access.
> > I setup authentication for postfix because I'm used to setting it up
> that
> > way, this was a force of habit.
> > I assumed postfix would run just fine if I set postfix up the way I
> usually
> > set it up.
>
> It might. But "Internet Mail" is another name for Dante's Third Circle
> of Hell. (You must have committed a horrible felony to be sentenced
> to administer an Internet mail host. :-D That's just a joke, it
> sounds to me like your "usual setup" is quite solid and secure.)
> Anyway, any deviation from the setup described in virtualenv.html is a
> chance for everything to go wrong. "What I usually do" is just not
> deep enough knowledge if you're working with a new mail application
> with complex routing needs.
>
😅😅 Funniest thing I've read today.
I agree with your last sentence, needed to be reminded of that.
>
> > > What does "lsof -i @127.0.0.1 -i '@[::1]' | grep 24" output?
>
> > python3 1592922 mailman 27u IPv4 177685024 0t0 TCP
> localhost:8001 (LISTEN)
> > python3 1592949 mailman 27u IPv4 177685024 0t0 TCP
> localhost:8001 (LISTEN)
> > python3 1592950 mailman 27u IPv4 177685024 0t0 TCP
> localhost:8001 (LISTEN)
>
> The above are Mailman (actually, a gunicorn application) listening for
> connections to its REST API. These are HTTP listeners used by
> Postorius and HyperKitty (I guess those aren't running?), not for email.
>
Postorius and HyperKitty are running.
> > postgres 3568035 postgres 5u IPv6 160224851 0t0 TCP
> localhost:postgresql (LISTEN)
> > postgres 3568035 postgres 6u IPv4 160224852 0t0 TCP
> localhost:postgresql (LISTEN)
>
> These are PostgreSQL listening for database connections.
>
> > python3 1592924 mailman 22u IPv6 177685051 0t0 TCP
> localhost:48638->localhost:postgresql (ESTABLISHED)
>
> This is Mailman talking to Postfix.
> > postgres 1592954 postgres 9u IPv6 177686124 0t0 TCP
> localhost:postgresql->localhost:48498 (ESTABLISHED)
> > postgres 1592912 postgres 8u IPv6 160224857 0t0 UDP
> localhost:36787->localhost:36787
>
> + 21 more copies of the above, differing only in the PID. Don't know
> what's going on with those, not a PostgreSQL wizard. I probably
> should have specified TCP in the lsof command, UDP applications are
> either trivial or they are black magic.
>
> It appears that *nothing* is listening on or connected to port 24.
> I'm not sure how that works. I'm guessing that when mail was getting
> to Mailman, Mailman had opened port 24 and Dovecot was cut out of the
> loop (maybe check logs for Dovecot complaining it couldn't bind to
> port 24?)
>
The logs don't show Dovecot complaining about binding to port 24, the only
thing.
This is one that's consistent with Dovecot in the logs;
157322 Feb 4 12:27:02 lists postfix/cleanup[1593312]: AC7381A8978:
message-id=<20260204202702.AC7381A8978(a)lists.sugarlabs.org>
157323 Feb 4 12:27:02 lists postfix/bounce[1587396]: A26F71A8977: sender
non-delivery notification: AC7381A8978
157324 Feb 4 12:27:02 lists postfix/qmgr[1593314]: AC7381A8978: from=<>,
size=3129, nrcpt=1 (queue active)
157325 Feb 4 12:27:02 lists postfix/local[1593315]: AC7381A8978: passing <
mailman(a)lists.sugarlabs.org> to transport=lmtp
157326 Feb 4 12:27:02 lists dovecot: lmtp(1600482): Connect from 127.0.0.1
157327 Feb 4 12:27:02 lists postfix/qmgr[1593314]: A26F71A8977: removed
157328 Feb 4 12:27:02 lists postfix/lmtp[1593316]: AC7381A8978: to=<
mailman(a)lists.sugarlabs.org>, relay=localhost[127.0.0.1]:24, delay=0.04,
delays=0.02/0/0/0.01, dsn=5.1.1, status=bounced (host localhost[127.0.0.1]
said: 550 5.1.1 <mailman(a)lists.sugarlabs.org> User doesn't exist:
mailman(a)lists.sugarlabs.org (in reply to RCPT TO command))
157329 Feb 4 12:27:02 lists dovecot: lmtp(1600482): Disconnect from
127.0.0.1: Logged out (state=READY)
Which we've established that mailman doesn't know how to deliver to local
users.
>
> > > > [mta] lmtp_host: *.*.*.*
> > >
> > > That should be 127.0.0.1. Mailman should not be listening on network
> > > interfaces other than the loopback (localhost).
> >
> > Yes, it is.
>
> OK
>
> > > > [mta] lmtp_port: 24
> > >
> > > This can't work if Dovecot is configured to use the same port, as is
> > > Dovecot's default.
>
> > I've stopped the service.
>
> OK
>
> > > > [mta] outgoing: mailman.mta.deliver.deliver
> > > > [mta] smtp_host: 127.0.0.1
> > > > [mta] smtp_port: 25
> > >
> > > Those are normal.
> > >
> > > > [mta] smtp_pass: **************
> > > > [mta] smtp_user: mailman
> > >
> > > This is going to cause Mailman to try to authenticate to Postfix.
> > > That's OK if it's what you want and Postfix / saslauthd has been
> > > properly configured to accept Mailman's credentials. Otherwise,
> > > they must be left empty to disable authentication.
>
> > Okay.
>
> > > > The Message-ID being the way it is doesn't seem like a
> > > > configuration issue, if it is then how do I look into it?
> > >
> > > Probably you don't have the full hostname in Postfix's myorigin or
> > > myhostname parameter.
> >
> > I do have the full hostname there, I also have it in /etc/mailname.
>
> I don't understand it then. Perhaps the mail composition client is
> providing the Message-ID.
>
Could be, doesn't seem to be an issue at the moment so I won't worry about
it.
>
> > > If not, I would guess that's a spammer who is somehow spoofing
> > > the PTR record.
>
> > The log entry before that warned of the same, they're definitely a
> spammer.
> > Relay access is denied though, so no mails are going through.
>
> OK.
>
> > No, I mean smtps.
>
> Same problem except the port that Mailman is *not* using is 465.
>
Yes, earlier when I set it up I had set the port for Mailman to use as 465.
>
> > > I don't think permit_auth_destination does what you think it does.
> > > Here's the Postfix doc:
> > >
> > > permit_auth_destination
> > > Permit the request when one of the following is true:
> > >
> > > - Postfix is a mail forwarder: the resolved RCPT TO domain
> > > matches $relay_domains or a subdomain thereof, and the address
> > > contains no sender-specified routing (user@elsewhere@domain),
> > > - Postfix is the final destination: the resolved RCPT TO domain
> > > matches $mydestination, $inet_interfaces, $proxy_interfaces,
> > > $virtual_alias_domains, or $virtual_mailbox_domains, and the
> > > address contains no sender-specified routing
> > > (user@elsewhere@domain).
> > >
> >
> > I did read the documentation before adding it, I definitely
> misunderstood
> > what it meant, and after looking at the old
> > documentation, it makes sense not to add it.
> >
> > Could you help me understand what the above means?
>
> I gotta run now, I'll come back to it later. You're probably asleep,
> anyway. :-)
>
> --
> GNU Mailman consultant (installation, migration, customization)
> Sirius Open Source https://www.siriusopensource.com/
> Software systems consulting in Europe, North America, and Japan
--
Ibiam Chihurumnaya
ibiamchihurumnaya(a)gmail.com
1 month, 1 week
Re: Mailman, etc. upgrade woes and persistent bugs
by Stephen J. Turnbull
Andreas Barth writes:
> * Stephen J. Turnbull (turnbull.stephen.fw(a)u.tsukuba.ac.jp) [210217 09:54]:
> > Right. So what you mean is "ok, we have group permissions, and maybe
> > someday there will be a case for ACLs down to arbitrary subsets of the
> > user population," but you don't have a concrete use case for arbitrary
> > ACLs at the moment.
> I definitly have a usecase for that, but: currently other priorities :)
> (I had to do an ad-hoc takeover of lots of infrastructure in my spare
> time within an volunteer organisation including mailing lists, so
> getting wishlist itmes done is not my top priority at the moment.)
>
> Just from the top of my head, for example I want to differentiate
> between private, member and public lists. So if a member logs in, they
> see more lists than a non-member. This is currently not possible, not
> too big a deal (all member lists are currently configured as private)
> but of course would be nice. Also archiv access should be aligned with
> these permissions.
This would not be worth doing with generic ACLs from Postorius, unless
the list had (many) fewer than 100 users. What I'd end up doing (and I
bet you would too) is write a 15-line mailmancli script that extracted
membership information and then created the ACLs. But you'd still
have to modify the Django templates, I suspect, and perhaps database
schema.
Essentially the same 15 lines could be put into Mailman itself without
ACLs, and checking ACL membership is not going to be much, if at all,
faster than checking list membership.
> Of course the appropriate split between private changes and public
> changes is also something which might need some discussions.
Not sure what you mean by private and public here, but if you mean
which changes we should maintain and which changes you should maintain
locally, I haven't seen anything in this thread that we wouldn't want
to offer all Mailman 3 users. From Abhilash's and Mark's posts, I
believe they feel that way too. It's just a question of whether a
very general implementation is needed/worthwhile.
Steve
5 years
Re: Emergency moderation and clearing out a lot of held messages
by Abhilash Raj
On 12/14/22 03:31, Mark Sapiro wrote:
> On 12/13/22 10:38, Dan Caballero wrote:
>> Hi Mark,
>>
>> I ran the commands and it took about 35 seconds for the loop to run.
>>
>> Here's the result.
>>
>>>>> count
>> 4168
>
>
> So that's at least part of the issue. How does 35 seconds compare to the
> length of time to process one moderated message through Postorius?
>
Probably should've read more of the thread before previous reply ;-)
So, the issue _could_ be due to latency to the remote database vs local
database that we have on m.p.o. Mailman3 is not very efficient when it
comes to total no. of database queries done per operation, which is
something I have been (very slowly) tracking and fixing on a
case-by-case basis.
So, for example, handling 1 held message in m.p.o made 1.1k postgres
database calls in total and even though they were each only few hundred
micro-seconds each, the total added up to roughly ~2sec avg response
time on the handle message API endpoint. If you handle more than 1, it
scales linearly.
My suspicion is that in your case Dan, due to a remote instance, the
latency per call is higher (maybe in the order of few or 10s of
milliseconds is my guess?) and then depending on the total no. of
entries you have in pendings and pendedkeyvalue tables (with some
filters, not _all_ entries), it adds up to a high value.
The no. of database calls here is of order n^2, where n is the entries
in pendings of type "held message" and "data" and their respective
linked relationships in pendedkeyvalue table.
My MR makes it such that we don't need to scan all entries of
"type"="data", so that part will become constant time. And the pending
of type "held message" will be limited to no. of held messages in a
single MailingList(compared to all mailing lists like today), so it will
help depending on the distribution of held messages in various lists. I
am also exploring ways to make that 2nd query also constant time.
--
thanks,
Abhilash Raj (maxking)
3 years, 3 months
Re: MM3 Postfix FROM header issue
by eboltz＠lhtservices.com
Mark,
Thanks for the assistance thus far.
When I create a new Domain within Postorius. If I set the Mail Host to 'example.com' and the Alias Domain to 'lists.example.com', create a new list such as 'staff_test', it'll default to the 'example.com' domain (which is the real domain). Then when I subscribe, I'll get an email from 'staff_test(a)example.com', when I want it to be from 'staff_test(a)lists.example.com' instead, also if I reply back to the 'staff_test(a)example.com' then I"ll get a bounce-back that it doesn't exist, but if I send to 'staff_test(a)lists.example.com' it'll go to the newly created list. I'd like all defaults to be '@lists.example.com', if that makes sense.
Here are the settings of my postfix_lmtp and postfix_vmap
# AUTOMATICALLY GENERATED BY MAILMAN ON 2023-12-17 20:05:34
#
# This file is generated by Mailman, and is kept in sync with the binary hash
# file. YOU SHOULD NOT MANUALLY EDIT THIS FILE unless you know what you're
# doing, and can keep the two files properly in sync. If you screw it up,
# you're on your own.
# Aliases which are visible only in the @lists.example.com domain.
staff_test(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-bounces(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-confirm(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-join(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-leave(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-owner(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-request(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-subscribe(a)lists.example.com lmtp:[127.0.0.1]:8024
staff_test-unsubscribe(a)lists.example.com lmtp:[127.0.0.1]:8024
# AUTOMATICALLY GENERATED BY MAILMAN ON 2023-12-17 20:05:34
#
# This file is generated by Mailman, and is kept in sync with the binary hash
# file. YOU SHOULD NOT MANUALLY EDIT THIS FILE unless you know what you're
# doing, and can keep the two files properly in sync. If you screw it up,
# you're on your own.
# Virtual mappings for the @example.com domain.
staff_test(a)example.com staff_test(a)lists.example.com
staff_test-bounces(a)example.com staff_test-bounces(a)lists.example.com
staff_test-confirm(a)example.com staff_test-confirm(a)lists.example.com
staff_test-join(a)example.com staff_test-join(a)lists.example.com
staff_test-leave(a)example.com staff_test-leave(a)lists.example.com
staff_test-owner(a)example.com staff_test-owner(a)lists.example.com
staff_test-request(a)example.com staff_test-request(a)lists.example.com
staff_test-subscribe(a)example.com staff_test-subscribe(a)lists.example.com
staff_test-unsubscribe(a)example.com staff_test-unsubscribe(a)lists.example.com
2 years, 2 months