Re: how to integrate subscription into static jekyll website
by Stephen J. Turnbull
Hagen Bauer writes:
> Has anybody tried to hide the direct api access behind a nginx reverse
> proxy url?
Sounds straightforward.
I doubt anybody's done it, though, because you still face the problem
that anybody with that access is a site manager, with unclear limits
on their power outside of Mailman. (Shouldn't be much they can do
outside of Mailman, but I wouldn't bet my host on it.) In many cases,
the real site manager is also the host admin, so they have sufficient
access to a shell account. There's no need for her to have a proxy;
why make it available to people who are supposed to have less power?
Note I'm *not* saying this *never* makes sense. I'm just saying that
in a lot of cases it won't buy you very much because Mailman isn't set
up to provide granular authorization of REST endpoints.
> on the Webseite I would have a url like this
>
> mailmanserver.tld/subcsribe/user(a)mail.com
>
> And within the nginx proxy configuration on the server this would
> translate to
>
> localhost:80xx/"mailman-restapi-with-credentials"user(a)mail.com
First of all, the address to subscribe is a variable, so at a bare
minimum your nice static site has to provide a form, which isn't very
static. I guess you could put the URL to the nginx proxy as the
action URL in the form, but then the proxy needs to deal with a
form-encoded query string to translate to the appropriate REST
endpoint.
I suppose you could also have the user type the URL including the
email address into the browser address bar, but Abhilash's idea of
providing a mailto:subscribe@mailmanserver.tld URL seems more
convenient. The mailto URL also solves the problem of confirming
ownership of the mailbox, since that logic is built in to the
subscribe-by-mail code.
Second, if your proxy is configured very carefully, you might be able
to restrict this to only the exact operations you want to expose, here
subscription. However, although the email interface implements
verification by providing a one-time token that must be read from the
mailbox of the address (and so proves ownership of the mailbox), I
don't think the REST API implements operations like that. IIRC that
is implemented in Postorius. (Abhilash?) If my memory is correct,
you would also need to implement that, and that's definitely not static.
> So the public does not see the credentials. And if my nginx host
> config is hacked I have a problem anyway.
I can't speak for Abhilash, but what would worry me is not that nginx
is already hacked, but rather that if not properly configured, it
might be possible for someone to access REST endpoints you don't want
exposed through this interface, and maybe even to other parts of the
webserver. To give a simple example, consider a variant on the
ancient Apache traversal bug:
mailmanserver.tld/subscribe/../unsubscribe/user(a)mail.com
Probably nginx and/or Mailman and/or your proposed proxy config are
already capable of stopping this particular well-known attack, but
undoubtedly there are more sophisticated attacks that I don't know
about.
6 years, 11 months
Re: Problems after update to mailman 3.3.7
by Guillermo Hernandez (Oldno7)
El 6/12/22 a las 15:59, Danil Smirnov escribió:
> Hi Guillermo,
>
> We've faced the same issue and already reported it:
> https://gitlab.com/mailman/mailman/-/issues/1044
>
I see. Thanks, Danil, for the confirmation that I'm not alone in this..
> I would not recommend anyone using MySQL/MariaDB database engine upgrade to
> the latest version yet.
Too late for that... Doy you know what components could I downgrade to
avoid this? SQLAlchemy? Which versions of which packages could I
downgrade to work prpoerly?
Again, thanks a lot
> With my best regards,
> Danil Smirnov
> Mailman3.com
>
> On Tue, Dec 6, 2022 at 4:32 PM Guillermo Hernandez (Oldno7) via
> Mailman-users<mailman-users(a)mailman3.org> wrote:
>
>> I've getting the most strange behaviour after update to last releases.
>> In one of my mailman3 servers, the messages doesnt reach for some lists
>> (the five last ones)
>>
>> the update have not showed any error but in the var/data for the lmtp
>> and vmap postfix files mailman does not create aliases for that last 5
>> lists. Any message addressed to them is returned with a "no such user"
>> error.
>>
>> If I create a new list the system or restart mailman runners it shows
>> this error:
>>
>> Generating MTA alias maps
>> /usr/local/lib/python3.9/site-packages/pymysql/connections.py:799:
>> UserWarning: Previous unbuffered result was left incomplete
>> warnings.warn("Previous unbuffered result was left incomplete")
>>
>> I did a CHECK TABLE on mailinglists table but all seems ok.
>>
>> I tried reinstalling PyMySQL, but has no effect. (well, I tried
>> reinstalling all)
>>
>> It seems to me that something is preventing creation of that aliases, an
>> then the that last five lists are unreachable.
>>
>> I have no clue of what to do now and I'm driving crazy.
>>
>> Some direction to look for will be much apreciated.
>>
>> Thanks in advance.
>>
>> P.S.: I can access via postorius/hyperkitty to the archives of the lists
>> that are unreacheable.
>>
>>
>> --
>>
>> ___________________________________________
>> Mailman's content filtering has removed the
>> following MIME parts from this message.
>>
>> Content-Type: image/png
>> Name: firma-GHP-emails.png
>>
>> Replaced multipart/alternative part with first alternative.
>> _______________________________________________
>> Mailman-users mailing list --mailman-users(a)mailman3.org
>> To unsubscribe send an email tomailman-users-leave(a)mailman3.org
>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>> Archived at:
>> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>>
>> This message sent todanil(a)smirnov.la
>>
> _______________________________________________
> Mailman-users mailing list --mailman-users(a)mailman3.org
> To unsubscribe send an email tomailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at:https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/mess…
>
> This message sent toguillermo(a)querysoft.es
>
--
___________________________________________
Mailman's content filtering has removed the
following MIME parts from this message.
Content-Type: image/png
Name: firma-GHP-emails.png
Replaced multipart/alternative part with first alternative.
2 years, 10 months
Re: Mailman, etc. upgrade woes and persistent bugs
by Stephen J. Turnbull
No answer required, I think I understand the situation pretty well
now. But if I'm missing something, I would very much appreciate
criticism.
Andreas Barth writes:
> Well, currently these are exclusive-states, I'd rather see them as
> or-able-states.
Right, I understand that. I don't understand *why*.
There are real costs to making them or-able. There are constraints:
'enabled' shouldn't (mustn't?) be or-able with any 'disabled-by-foo'
state. Such invariants are finicky to code and tricky to maintain,
compared to a simple "mutually exclusive".
I guess you want to allow the or of any subset of disabled states, but
presumably then they need to individually resettable in order for the
admin to clear their disable without overriding a subscriber's
preference or a bounce record. That set of controls is more complex,
and requires additional mental effort on the part of anyone using
them. If there are restrictions on which states can be or-ed, that's
more coding and maintenance cost.
Then there's the question whether the subscriber can override the admin's
setting. If they can, why does the admin have this power in the first
place?
> > > - "disabled by member" (could be set by an admin as well of course)
> >
> > Currently this is determined by whether the logged in user is the
> > subscriber or not. It's an interesting idea to allow it to be set by
> > an admin. Then we could have the semantics that the only transition
> > allowed from this state would be to enabled, so that admins would know
> > the user had intentionally disabled and know not to enable without
> > permission.
>
> Exactly.
But if the admin should respect the DBS (disabled by subscriber)
setting, DBS|DBA (disabled by admin) is logically equivalent to DBS.
Are there use cases where the admin does not want to repect the
subscriber's choice, and enables it anyway? What are they?
> Or while importing from another server, and there are people who
> have receiving mails disabled, this is the state that could be used
> for it.
Why wouldn't they get the exact state from the original server? Or do
you mean from a non-Mailman server?
> I expect that state to be the most-often used when setting a state
> manually.
>
> For "who set that", I would consider having some kind of history as
> useful (and displaying "last set by user / $person" to the admin but
> not the user).
Why not to the subscriber? As I explain below, the subscriber at
least needs to be able to distinguish between DBA and DBB (disabled by
bounce).
Keeping history of who is additional complexity, consumes a minor
amount of resources, and is insufficient to the purpose. Assuming
that there are reasons for admins to disable/enable other than as a
courtesy to a subscriber, that reason is presumably important. Not only
will the subscriber not know, but one admin may forget, and there may be
multiple admins.[1]
I still don't see any reason for an admin to disable delivery other
than as a courtesy to the subscriber or to the bounce processor.
> > > - "disabled for administrative reasons" (only by admin but user
> > > should be able to see it)
> >
> > I don't see why a user would need to see the difference. Unless
> > you're suggesting that the user would not be able to reenable without
> > asking the admin? What is the use case for this?
>
> I'm suggesting exactly that, yes. Why? Well, worked with user for too
> long. ;)
>
> E.g. consider that seeing archives works only for subscribers. And the
> subscriber in question complains about too many mails but still
> re-enables getting mails. This is basically "an admin has blocked you
> from more mails but not unsubscribed you". Might be an corner case,
> but still. Might also be too unimportant.
This is possible. If it occurs less than once in a million years,
it's "too unimportant" to even think about.[2] Have you actually done
this, or know anybody who has?
Is it really different from "at request of subscriber"? Practically,
the only difference I can see is if such a subscriber should be
prevented from overriding DBA. Are you suggesting that?
> > > - "bounced" (user and admin could reset it but not set it)
> >
> > Reenable but not set "bounced" is the current situation.
>
> Except if someone sets the state to "disabled" and then undo this, the
> state is automatically cleared.
This is possible, but it's not clear what harm would be done. In most
cases, bounces are intermittent and clear themselves up, in which case
this is the right thing. But if the subscriber bounces again, they'll
get disabled again, only cost is a slight waste of resources.
Here's how I would assess the issues. In the descriptions below, I am
assuming that each state is the only "active" state.
1. DBS is probably a "vacation" or "post-only" setting. For that
reason it should be re-enabled only by the subscriber (or by an
admin at the request of the subscriber).
2. DBB is a "warning" and a minor convenience to the email system
(slightly reducing traffic to mailboxes that are expected to fail,
and perhaps avoiding some reputational damage to the list).
Re-enabling much of the time causes no damage, as bounce disables
are typically due to things like out of disk space, and are likely
fixed by the time the subscription is reenabled. Most of the rest
will be zombie mailboxes, which will rarely be reenabled, will
quickly be redisabled by bounce, and at worst cause mild
inconvenience. Finally, events like a Mailman upgrade and the
April 2014 DMARC fiasco will be pretty evident, and the damage
there has already been done, reenabling actually fixes it
(although it may not be effective if the systemic problem hasn't
been resolved).
3. DBA is interesting mostly to *subscribers*, who need to
distinguish DBA (which the subscriber can reenable freely) from
DBB (which warns the subscriber to get in touch with their
postmaster). It's comforting to subscribers who may be sure they
never disabled their subscription; otherwise, it's semantically
identical to DBS as far as I can see.
Am I missing something?
It seems to me that given 1-3, assuming a 4-state design:
0. It's never unreasonable to re-enable, except DBS.
1. In the DBA state, changing to either DBS or DBB causes no
problems. (Changing to DBB is unlikely, since no mail is being
sent. However, it's possible, due to the race between Postorius
and mail in transit, and due to spam and other spoofing of the
list domain.)
2. In either of the other two states, we should never change to DBA.
3. In either of the other two states, if the other event occurs, you
would want the state to be DBS|DBB. I contend that if we
substitute DBS, little harm is done. If the subscriber reenables, it's
on them to fix the bounce problem if it reoccurs in any case.
Since the subscriber has deliberately set DBS, the admin should not
reenable without permission of the subscriber (I think we're mostly in
agreement on that).
I think this 4-state design is much easier to design, validate, and
maintain than the 16-state design you propose. If you're really
worried about DBS|DBB, then we could have a 5-state design where
enabling from DBS|DBB generates a warning that there may still be a
bounce problem on the subscription. Instead, it could leave the
subscription in state DBB, but that seems to me to be obnoxious, since
there's little likelihood of harm from reenabling, and in many cases
the bounce problem will have resolved itself in the meantime.
Regards,
Steve
Footnotes:
[1] There's also the question of GDPR obligations if personal reasons
(such as "violated code of conduct") are in the history.
[2] I'm not sure where we'd actually draw the line, probably on the
order of "once a year per thousand subscribers", that being every week
for say a campus-wide list at Ohio State University with 55,000 students.
4 years, 7 months
Re: Getting connection refused on postfix trying to connect to mailman-core:lmtp
by Dmitry Makovey
On 10/05/2017 07:20 PM, Abhilash Raj wrote:
> On Thu, Oct 5, 2017, at 05:36 PM, Dmitry Makovey wrote:
>> On 10/05/2017 03:20 PM, Abhilash Raj wrote:
>>> On Thu, Oct 5, 2017, at 03:08 PM, Dmitry Makovey wrote:
>>>>
>>>> I've got a setup where postfix runs inside one VM(container) and mailman
>>>> runs inside another one (maxking containers). I've wired everything
>>>> according to docs yet I'm getting:
>>>>
>>>> postfix/lmtp[266]: 66A72800A87: to=<somelist(a)lists.here.stanford.edu>,
>>>> relay=none, delay=0.5, delays=0.48/0.01/0/0, dsn=4.4.1, status=deferred
>>>> (connect to mailman-01.stanford.edu[1.2.3.4]:8024: Connection refused)
>
> I found this in your settings:
>
> [mta] lmtp_host: mailman-01.stanford.edu
> [mta] lmtp_port: 8024
>
>
> And the log message above.
>
> I believe that the LMTP runner died because it wasn't able to bind to
> `mailman-01.stanford.edu`, which I am assuming is the hostname assigned
> to the host running these containers.
>
> `MM_HOSTNAME` env variable in the docker containers should be something
> that the process inside mailman-core container can bind to and can be
> reached by postfix (which can run either on host or on another
> container). (Now that I read it myself, I agree that the name of the
> variable sounds not-so-intuitive.)
thank you so much for the hints! I've changed docker-compose to include
MM_HOSTNAME variable *and* made sure that for the mailman-core I've got:
services:
mailman-core:
hostname: mailman-01
domainname: stanford.edu
...
environment:
...
MM_HOSTNAME: mailman-01.stanford.edu
that solved the issue while keeping the mentioned mailman config. Is
that what you've had in mind? I was looking for a quick hack, but still
would like to find out proper solution if that isn't the one.
> The default configuration (and docker-compose.yaml) sets this value to
> the IP Address of the container (172.19.199.2), which is reachable from
> the host. If you set this value to whatever IP the mailman-core is
> assigned and re-create the containers (or just re-start and run `mailman
> aliases` in mailman-core to re-generate transport_maps), it would work
> out.
if I understand above correctly that means semi-manual mangling of
postfix aliases file which I'd rather not do. Using above technique
mailman-core does generate proper aliases while binding to the
appropriate IP. kind of icky but seems to work.
>
> Also, I see your docker-compose.yaml configuration (`MAILMAN_CORE_IP:
> 172.19.199.2`) is not consistent with your output of `mailman
> conf`([webservice] hostname: 172.19.199.5`). I am not sure how that
> happened though, just wanted to point it out. This *might* cause the
> rest runner to die to and thus Postorius/HK wouldn't work.
right, some of the IPs keep on popping at me and I've got to look for
them - I intentionally removed direct IP assignment in
docker-compose.yaml yet I get the feeling that someplace there's another
IP assignement I've missed.
> Hope that was helpful!
very much so! thanks for your quick responses - provided my existing
deadlines I value them that much more :)
--
Sr System and DevOps Engineer SoM IRT
8 years
Re: where to configure max_days_to_hold on MM3 GUI interface?
by Abhilash Raj
On Sat, Feb 1, 2020, at 5:27 AM, Shashikanth Komandoor wrote:
> Hello Mark,
>
> Thanks for helping me for moving ahead in my project.
>
> Troubling you again, I would like to put another request at you.
>
> As per your suggestion, I am trying to configure my list
> configuration through REST API for specially the parameter like
> max_days_to_hold as it is not available through interface and I am
> following the official links of mailman 3 i.e
> https://mailman.readthedocs.io/en/latest/src/mailman/rest/docs/listconf.html
>
> But I am getting "name is not defined" sort of errors as below:
>
>
>
>
>
> *>>> dump_json('http://localhost:9001/3.0/lists/
> <http://localhost:9001/3.0/lists/>','ant(a)example.com/config
> <http://ant@example.com/config>',dict(display_name='My
> List'),'PATCH')Traceback (most recent call last): File "<stdin>", line 1,
> in <module>NameError: name 'dump_json' is not defined>>>*
>
> Actually I am new to REST API but hope this is the only issue that
> is preventing me to go ahead.
>
> Could you please help me by saying how should I handle this so that
> I should be able to configure *max_days_to_hold* of a list or multiple ?
>
dump_json is basically a Python function which sends the request to the URL you add as the first parameter (GET, if any data isn't specified and uses POST if data is specified or explicitly define the PATCH or method like you have above) and prints the json response.
You can easily do the same using curl with the right authorization headers, with something like:
$ curl -u restadmin:restpass -X PATCH \
http://localhost:8001/3.1/lists/somelist.example.com/config -d display_name='Patched Name'
Note that "restadmin:restpass" is the username password that you have set in your mailman.cfg.
If you do want to use Python, you can just use a regular PATCH request using requests or standard library urllib3.
Finally, dump_json is documented here: https://mailman.readthedocs.io/en/latest/src/mailman/docs/documentation.htm…
> On Wed, Jan 29, 2020 at 11:20 PM Mark Sapiro <mark(a)msapiro.net> wrote:
>
> > On 1/28/20 8:42 PM, Shashikanth Komandoor wrote:
> > >
> > > But I could see that parameter value configurable on MM2 GUI
> > > interface in General Options but I don't see the same on the MM3 GUI
> > > interface.
> >
> >
> > There are many core settings not (yet) exposed in Postorius.
> > max_days_to_hold. is one of them. It can be set via `mailman shell` or
> > via the REST API
> > <
> > https://mailman.readthedocs.io/en/latest/src/mailman/rest/docs/rest.html#re…
> > >.
> >
> > --
> > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> > San Francisco Bay Area, California better use your sense - B. Dylan
> > _______________________________________________
> > Mailman-users mailing list -- mailman-users(a)mailman3.org
> > To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >
>
>
> --
> Thanks & Regards,
> Shashi Kanth.K
> 9052671936
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>
--
thanks,
Abhilash Raj (maxking)
5 years, 8 months
Re: mailman-web migrate fails with sqlite3.IntegrityError: UNIQUE constraint failed: account_emailaddress.email
by Stefan Bauer
Dear Mark,
thank you for your time.
After patching your script according to
https://github.com/maxking/docker-mailman/issues/729#issuecomment-2447542238
It fails with
/opt/mailman/venv/data# /usr/local/bin/UC_fix
/opt/mailman/venv/data/mailman3web.db
Traceback (most recent call last):
File "/usr/local/bin/UC_fix", line 77, in <module>
c.execute("delete from account_emailaddress where email = %s;",
sqlite3.OperationalError: near "%": syntax error
for reference:
67 for id, emails in d.items():
68 ems = find(emails)
69 if ems:
70 em1, em2 = ems
71 if em1.lower() == em1:
72 c.execute("delete from account_emailaddress where email =
%s;",
73 [em2])
74 c.execute("""update account_emailaddress set "primary" = 't'
75 where email = %s;""", [em1])
76 elif em2.lower() == em2:
77 c.execute("delete from account_emailaddress where email =
%s;",
78 [em1])
79 c.execute("""update account_emailaddress set "primary" = 't'
80 where email = %s;""", [em2])
81 else:
82 print(f'No LC email for {ems}')
However the sql connection itself works, and the select also reports
addresses:
sqlite> select email, user_id from account_emailaddress;
...
John.Doe(a)my.domain|85
john.doe(a)my.domain|85
.....
Mike.Doe(a)my.domain|96
mike.doe(a)my.domain|96
....
Is there something wrong on my side?
Thank you.
Stefan
Am Mo., 3. März 2025 um 18:40 Uhr schrieb Mark Sapiro <mark(a)msapiro.net>:
> On 3/3/25 07:52, Stefan Bauer via Mailman-users wrote:
> >
> > trying to update mailman3 to lastest version, we fail during
> >
> > mailman-web migrate with:
> >
> > Operations to perform:
> > Apply all migrations: account, admin, auth, contenttypes,
> django_mailman3,
> > django_q, hyperkitty, postorius, sessions, sites, socialaccount
> > Running migrations:
> > Applying account.0006_emailaddress_lower...Traceback (most recent call
> > last):
> > File
> >
> "/opt/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> > line 89, in _execute
> > return self.cursor.execute(sql, params)
> > File
> >
> "/opt/mailman/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py",
> > line 328, in execute
> > return super().execute(query, params)
> > sqlite3.IntegrityError: UNIQUE constraint failed:
> account_emailaddress.email
>
>
> This issue is due to changes in Django allauth to prevent creating
> multiple addresses differing only in case. The problem is if you have
> such multiple addresses in your account_emailaddress table, this
> migration fails. There is a script at
> https://www.msapiro.net/scripts/UC_fix that will fix this. You may need
> to adjust the shebang and replace psycopg2 with sqlite3.
>
> See https://github.com/pennersr/django-allauth/issues/3019 perhaps
> starting at
>
> https://github.com/pennersr/django-allauth/issues/3019#issuecomment-1748028…
>
> > We found a workaround to fake the migration with
> >
> > migrate --fake
>
> You should fix the issue with the DB and run the migrations.
>
> > Could this be some duplicate users with same mail-address?
>
> This is not related to Mailman users. It is Django users.
>
> --
> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> San Francisco Bay Area, California better use your sense - B. Dylan
>
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at:
> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>
> This message sent to cubewerk(a)gmail.com
>
7 months
Re: Getting rid of held messages from the command line
by Mark Sapiro
On 03/16/2018 09:01 AM, Aymeric Mansoux wrote:
>
> However, while I was testing this happily on a list that was not affected with the JSON error, I finally tested discard_held_messages() on one of the problematic list, and I get the exact same problem as when trying to view the moderation queue API:
>
>>>> discard_held_messages_queue('xxx(a)xxx.xxx')
> Traceback (most recent call last):
> File "<console>", line 1, in <module>
> File "<console>", line 6, in discard_held_messages_queue
> File "/usr/local/lib/python3.5/dist-packages/mailman/app/moderator.py", line 97, in handle_message
> key, msgdata = requestdb.get_request(id)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request
> result.data_hash, expunge=False)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm
> value = json.loads(keyvalue.value)
> File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
> obj, end = self.scan_once(s, idx)
> json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
>
> With that said, even though I cannot pass the discard action, I can export the queue without problem, so I will send you off-list a sample, because I am not sure what I should be looking for.
Unfortunately, that didn't help. I was able to post all the messages in
your sample to a test list and even with all the messages held, I could
view them and discard them in postorius.
This may not be too surprising as looking at the traceback, the
exception is thrown in trying to delete the pending confirmation from
the pending database. The issue is when a message is held an entry is
made in the pending database with a confirmation token that can be used
for example to approve or discard the message by email.
There is an issue that affects MySQL databases only in that prior to the
merge of <https://gitlab.com/mailman/mailman/merge_requests/333> values
in the pendedkeyvalue table could be truncated (or exceptions thrown
depending on MySQL settings). That appears to be the issue here.
The fix was only merged in January and is not in Mailman 3.1.1.
> Finally, is there another way to get rid of these messages? Are there consequences if I would use message_store.delete_message(message_id) instead of handle_message(m, req.id, Action.discard)?
As I said, I think the issue is with the pendedkeyvalue table, not with
the message itself. The table contains entries which are pickled
versions of various things. A big issue is rule_misses. For one of your
messages, the rule_misses list is
['dmarc-mitigation', 'no-senders', 'approved', 'emergency', 'loop',
'banned-address', 'member-moderation']
this gets pickled into
b'\x80\x03]q\x00(X\x10\x00\x00\x00dmarc-mitigationq\x01X\n\x00\x00\x00no-sendersq\x02X\x08\x00\x00\x00approvedq\x03X\t\x00\x00\x00emergencyq\x04X\x04\x00\x00\x00loopq\x05X\x0e\x00\x00\x00banned-addressq\x06X\x11\x00\x00\x00member-moderationq\x07e.'
which ultimately gets stored in the pendedkeyvalue table as a value for
'_pck_rule_misses' which is
'"\u0080\u0003]q\u0000(X\u0010\u0000\u0000\u0000dmarc-mitigationq\u0001X\n\u0000\u0000\u0000no-sendersq\u0002X\b\u0000\u0000\u0000approvedq\u0003X\t\u0000\u0000\u0000emergencyq\u0004X\u0004\u0000\u0000\u0000loopq\u0005X\u000e\u0000\u0000\u0000banned-addressq\u0006X\u0011\u0000\u0000\u0000member-moderationq\u0007e."'
Which is longer than the VARCHAR(255) MySQL field for that value prior
to <https://gitlab.com/mailman/mailman/merge_requests/333>.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
7 years, 6 months
Re: UnicodeEncodeError: 'ascii' codec can't encode character
by Stephen J. Turnbull
Marvin Gülker writes:
> Am 10. Dezember 2019 um 17:09 Uhr +0900 schrieb Stephen J. Turnbull:
> > You're the "it's 2019, let's use SMTP UTF8 everywhere" guy, right?
>
> No, I'm not that guy.
I'm sorry about that. I knew I didn't know for sure, but not being
able to get at the moderation queue seemed urgent enough to be worth
the chance of being wrong.
> I have not used it, but it turns out that it is enabled by
> default. I can turn it off and see if that makes any difference in
> the future. Testing with telnet, Postfix indeed announced "250
> SMTPUTF8" on EHLO.
>
> > If you're right about the umlaut being the trigger, the first
> > thing I'd look at is a feature-negotiation problem in the MTA
> > delivering to Mailman. I'm pretty sure our LMTP does not offer
> > the SMTP UTF8 feature.
>
> In that case the problem should go away with disabling SMTPUTF8, I
> suppose?
Well, since it's predicated on a bug in Postfix, I wouldn't bet on it,
especially since Postfix probably dominates Mailman 3 installations.
But it's cheap to test, and since you don't explicitly want SMTP UTF8
at this point, little harm in turning it off. *If* there is a bug,
that will prevent the problem in the future, but it wouldn't fix the
problem in the held messages queue.
> > I would guess this is in the shunt queue. If there's only one
> > file there, you can just delete it.
This was a bad guess. I'm not sure why it isn't in the shunt queue
(it shouldn't have been able to escape into the main rule chains with
that bad breakage in the header), but Mailman doesn't try to moderate
shunted messages, that's the whole point of the shunt queue -- they're
out of the way.
> There was a whole bunch of files in that directory. I've taken a look
> several of them, and as they were all spam, I've taken the liberty to
> delete them all (this list is so low volume that I know all the senders
> personally anyway).
A tiny bit of good came of it, at least.
> However, even after restarting mailman, the 500 error persists when
> visiting the held messages page in postorius.
>
> I've also looked into all the other queue directories, they were all
> empty.
I guess it's in the MessageStore, then, which is by default in
/var/tmp/mailman/messages/. (Defined in schema.cfg.) I guess it's a
standard qfile that you can examine with "mailman qfiles
/path/to/file". (I've never actually looked at a "raw" held message
file in Mailman 3, haven't had a failure of this type.) If not, you
can unpickle it with Python (if that makes no sense to you, ask; I'm
running out of steam and need sleep).
> > Assuming you're right about your name being the trigger, U+FFFD
> > REPLACEMENT CHARACTER is not in your name.
I'm starting to wonder about this. It's still possible, but if you
don't deliberately use SMTP UTF8 then your mail client probably
doesn't either, and equally possibly it's a different message causing
the problem (something spammy).
5 years, 10 months
Re: Hyperkitty CPU usage
by Alain Kohli
I'm running a custom image which is based on an older version of the one
here: https://github.com/maxking/docker-mailman. I attached it below.
But I separated postorius and hyperkitty, so hyperkitty is running in
its own container. I'm deploying the image with a plain 'docker run'
behind nginx. I made fulltext_index persistent now, but it didn't get
populated with anything yet. I don't really have an error traceback
because there is never an error thrown. The only thing with some content
is uwsgi-error.log, which you can find below. I'm also still getting the
"A string literal cannot contain NUL (0x00) characters." messages. I
also noticed that it takes incredibly long for the webinterface to load
(several minutes) even though there doesn't seem to be any process
consuming notable resources apart from the minutely job.
Funnily enough, I have the exact same image deployed on a second server
as well for testing. On that one everything works fine. The only
difference is that on the problematic one I have a lot more mailing
lists/archives and that I imported them from mailman2. Could something
have gone wrong during the import? I used the regular hyperkitty_import
command.
uwsgi-error.log:
*** Starting uWSGI 2.0.18 (64bit) on [Sat Apr 27 22:50:17 2019] ***
compiled with version: 6.4.0 on 27 April 2019 22:48:42
os: Linux-4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19)
nodename: hyperkitty.docker
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /home/hyperkitty
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 82
setuid() to 82
chdir() to /home/hyperkitty
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:8081 fd 8
uwsgi socket 1 bound to TCP address 0.0.0.0:8080 fd 9
Python version: 3.6.8 (default, Jan 30 2019, 23:54:38) [GCC 6.4.0]
Python main interpreter initialized at 0x55dfaa41c980
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
[uwsgi-cron] command "./manage.py runjobs minutely" registered as
cron task
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" registered
as cron task
[uwsgi-cron] command "./manage.py runjobs hourly" registered as cron
task
[uwsgi-cron] command "./manage.py runjobs daily" registered as cron task
[uwsgi-cron] command "./manage.py runjobs monthly" registered as
cron task
[uwsgi-cron] command "./manage.py runjobs weekly" registered as cron
task
[uwsgi-cron] command "./manage.py runjobs yearly" registered as cron
task
mapped 208576 bytes (203 KB) for 4 cores
*** Operational MODE: threaded ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter
0x55dfaa41c980 pid: 1 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 40, cores: 4)
Sat Apr 27 22:50:18 2019 - [uwsgi-cron] running "./manage.py runjobs
minutely" (pid 45)
[uwsgi-daemons] spawning "./manage.py qcluster" (uid: 82 gid: 82)
22:50:21 [Q] INFO Q Cluster-47 starting.
22:50:21 [Q] INFO Process-1:1 ready for work at 59
22:50:21 [Q] INFO Process-1:2 ready for work at 60
22:50:21 [Q] INFO Process-1:3 ready for work at 61
22:50:21 [Q] INFO Process-1:4 ready for work at 62
22:50:21 [Q] INFO Process-1:5 monitoring at 63
22:50:21 [Q] INFO Process-1 guarding cluster at 58
22:50:21 [Q] INFO Process-1:6 pushing tasks at 64
22:50:21 [Q] INFO Q Cluster-47 running.
22:59:31 [Q] INFO Enqueued 3403
22:59:31 [Q] INFO Process-1:1 processing [update_from_mailman]
22:59:33 [Q] INFO Processed [update_from_mailman]
Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs
quarter_hourly" (pid 73)
Sat Apr 27 23:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs
hourly" (pid 74)
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
with pid 73 exited after 64 second(s)
23:01:28 [Q] INFO Enqueued 3404
23:01:29 [Q] INFO Process-1:2 processing
[rebuild_mailinglist_cache_recent]
[uwsgi-cron] command "./manage.py runjobs hourly" running with pid
74 exited after 91 second(s)
Sat Apr 27 23:01:36 2019 - uwsgi_response_write_body_do(): Broken
pipe [core/writer.c line 341] during GET / (212.203.58.154)
OSError: write error
23:01:36 [Q] INFO Processed [rebuild_mailinglist_cache_recent]
Sat Apr 27 23:15:00 2019 - [uwsgi-cron] running "./manage.py runjobs
quarter_hourly" (pid 88)
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
with pid 88 exited after 4 second(s)
23:28:24 [Q] INFO Enqueued 3405
23:28:24 [Q] INFO Process-1:3 processing [update_from_mailman]
23:28:25 [Q] INFO Processed [update_from_mailman]
Sat Apr 27 23:30:00 2019 - [uwsgi-cron] running "./manage.py runjobs
quarter_hourly" (pid 96)
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
with pid 96 exited after 4 second(s)
23:44:40 [Q] INFO Enqueued 3406
23:44:40 [Q] INFO Process-1:4 processing [update_from_mailman]
23:44:41 [Q] INFO Processed [update_from_mailman]
Sat Apr 27 23:45:00 2019 - [uwsgi-cron] running "./manage.py runjobs
quarter_hourly" (pid 104)
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
with pid 104 exited after 4 second(s)
Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs
quarter_hourly" (pid 113)
Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs
hourly" (pid 114)
Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs
daily" (pid 115)
Sun Apr 28 00:00:00 2019 - [uwsgi-cron] running "./manage.py runjobs
weekly" (pid 116)
[uwsgi-cron] command "./manage.py runjobs quarter_hourly" running
with pid 113 exited after 55 second(s)
[uwsgi-cron] command "./manage.py runjobs weekly" running with pid
116 exited after 55 second(s)
00:01:36 [Q] INFO Enqueued 3407
00:01:36 [Q] INFO Process-1:1 processing
[rebuild_mailinglist_cache_recent]
[uwsgi-cron] command "./manage.py runjobs hourly" running with pid
114 exited after 99 second(s)
00:01:50 [Q] INFO Processed [rebuild_mailinglist_cache_recent]
00:04:52 [Q] INFO Enqueued 3408
00:04:52 [Q] INFO Process-1:2 processing [update_from_mailman]
00:04:54 [Q] INFO Processed [update_from_mailman]
Dockerfile:
FROM python:3.6-alpine3.7 # Add startup script to container COPY
assets/docker-entrypoint.sh /usr/local/bin/ # Install packages and
dependencies for hyperkitty and add user for executing apps. # It's
important that the user has the UID/GID 82 so nginx can access the
files. RUN set -ex \&& apk add --no-cache --virtual .build-deps gcc
libc-dev linux-headers git \postgresql-dev \&& apk add --no-cache
--virtual .mailman-rundeps bash sassc mailcap \postgresql-client
curl \&& pip install -U django==2.2 \&& pip install
git+https://gitlab.com/eestec/mailmanclient
\git+https://gitlab.com/mailman/hyperkitty@c9fa4d4bfc295438d3e01cd93090064d004cf44d
\git+https://gitlab.com/eestec/django-mailman3 \whoosh \uwsgi
\psycopg2 \dj-database-url \typing \&& apk del .build-deps \&&
addgroup -S -g 82 hyperkitty \&& adduser -S -u 82 -G hyperkitty
hyperkitty \&& chmod u+x /usr/local/bin/docker-entrypoint.sh# Add
needed files for uwsgi server + settings for django COPY
assets/__init__.py /home/hyperkittyCOPY assets/manage.py
/home/hyperkittyCOPY assets/urls.py /home/hyperkittyCOPY
assets/wsgi.py /home/hyperkittyCOPY assets/uwsgi.ini
/home/hyperkittyCOPY assets/settings.py /home/hyperkitty# Change
ownership for uwsgi+django files and set execution rights for
management script RUN chown -R hyperkitty /home/hyperkitty && chmod
u+x /home/hyperkitty/manage.py# Make sure we are in the correct
working dir WORKDIR /home/hyperkittyEXPOSE 8080 8081# Use stop
signal for uwsgi server STOPSIGNAL SIGINTENTRYPOINT
["docker-entrypoint.sh"]CMD ["uwsgi", "--ini",
"/home/hyperkitty/uwsgi.ini"]
On 4/27/19 7:58 PM, Abhilash Raj wrote:
> On Sat, Apr 27, 2019, at 9:40 AM, Alain Kohli wrote:
>> I have run "python manage.py rebuild_index" before, doesn't that do
>> clear_index as well? Apart from that, I run hyperkitty in a docker
>> container and didn't know fulltext_index should be persistent, so that
>> got deleted after every version update for sure.
> Which images are you using and how are you deploying them?
>
> You should persist fulltext_index, yes, and possibly logs if you need
> them for debugging later.
>
> Can you paste the entire error traceback?
>
>>
>> On 4/26/19 10:18 PM, Mark Sapiro wrote:
>>> On 4/26/19 11:14 AM, Alain Kohli wrote:
>>>> I see loads of "A string literal cannot contain NUL (0x00) characters."
>>>> messages, but I haven't found missing messages in the archives yet. Not
>>>> sure how that could be related, though. Apart from that I don't see
>>>> anything unusual. The other jobs (quarter_hourly, hourly, etc.) seem to
>>>> run and finish normally.
>>> Did you upgrade from a Python 2.7 version of HyperKitty to a Python 3
>>> version? The Haystack/Whoosh search engine databases are not compatible
>>> between the two and "A string literal cannot contain NUL (0x00)
>>> characters." is the symptom.
>>>
>>> You need to run 'python manage.py clear_index' or just remove all the
>>> files from the directory defined as 'PATH' under HAYSTACK_CONNECTIONS in
>>> your settings file (normally 'fulltext_index' in the same directory that
>>> contains your settings.py.
>>>
>> _______________________________________________
>> Mailman-users mailing list -- mailman-users(a)mailman3.org
>> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>>
6 years, 5 months
Re: Moderation action "None" is misleading
by Stephen J. Turnbull
Adding mailman-users(a)mailman3.org.
Technical suggestions should go to distro channels or to Mailman's
issue tracker: https://gitlab.com/mailman/mailman/-/issues/1176.
Further discussion of technical details is best done there.
Thomas Krichel writes:
> Stephen J. Turnbull writes
> > Debian has traditionally been quite slow to update Mailman,
>
> And now it has the addition issue that Debian testing python
> standard it 3.13, so the nntplib is no longer supported.
> Thus the current Mailman debian testing appears broken.
That's Debian's problem, as we currently don't support Python 3.13,
although of course we intend to. This is the only issue that came up
in a quick search of Mailman's issue tracker for Python 3.13 (but I
did not check the other subprojects such as Postorius). I conclude
that not a lot of work has been done on supporting Python 3.13 (rather
than the optimistic "there's nothing to be done"). From Mailman's
point of view, users have two excellent options for the forseeable
future: (1) don't gateway list <-> newsgroups (2) install Mailman into
a Python 3.9--3.12 virtual environment.
I'm sympathetic to the needs of the Debian developers, and at this
point for 99% of software it's perfectly reasonable to aim to use 3.13
in the next Debian release. Unfortunately for Mailman users on Debian
hosts that's not high on our list of priorities since we recommend a
virtual environment installation in any case.
At present there seem to be three ways forward for Debian packaging:
1. Remove NNTP support. (In honor of larsi, I'd rather not.)
2. Add nntplib.py from the Python 3.12 nntplib, as a stopgap until
option 3 can be implemented. I can't speak for other developers
but we might be willing to add it as a contrib module, especially
if that might make life easier for distro packagers. It's
suboptimal though since that absolutely has to be considered
techical debt that will need to be paid someday.
3. Port Mailman to pynntp (that may need to be Debian packaged, too),
which has seen a revival of activity and two feature releases
since 2023.
My inclination as a Mailman core developer is not to participate
actively in development of NNTP support in Mailman for Python 3.13+
unless we see an outpouring of support from site admins. However, I
would be happy to answer questions and work on Mailman-side issues
that are blocking or inconvenient for the NNTP work. Mark and
Abhilash would undoubtedly contribute as well, on a time-available
basis.
> I certainly would love to build an alternative Mailman Debian
> package that would be more up-to-date, but I don't want
> to do it on my own.
Debian and Python are compatible free software; just take the existing
package metadata, in a recent Mailman (presumably the most recent
release) drop in the most recent nntplib, and change all the import
statements. mailman/src/mailman/mta/nntplib.py seems like a plausible
location. IWBNI you could keep all Mailmanisms out of nntplib
(because of the GPL) so any bugfixes or improvements could be
available to the Python community.
I think you could probably be up and running VERY quickly with that
approach. Of course it wouldn't just slot into the next iteration of
Debian because they have stringent QA (or maybe that could go into
unstable?). I'm pretty sure it would be useful right away to a whole
lot of people who are willing to use a PPA.
Regards,
Steve
8 months, 1 week