On 6/26/23 11:53 AM, Ken Alker wrote:

Is the var/messages hierarchy also where all the archived emails are stored for future viewing via the web interface?

No. They are stored in the hyperkitty_email and hyperkitty_attachment tables in the database.

...

Also, there is a long thread at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/threa d/UWCUQYU7Q5X5LWNII57ZZ4DDXE4DEFX7/ and maybe others that might be relevant.

I have never used the mailman shell before, but based on the referenced thread I learned to do this (note that the thread also taught me how to rewrite it to narrow down to just the list with issues so I didn't delete held messages in other lists, if my case):

for verification:

 venv) mailman@speedy:~/web/logs$ mailman shell -l sbarc-list@lists.netlojix.com Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the sbarc-list@lists.netlojix.com mailing list >>> requests = IListRequests(m) >>> first = True >>> for request in requests.held_requests: ...     key, data = requests.get_request(request.id) ...     if first: ...             first = False ...             print(m.list_id) ...     print (f"""
... Sender: {data['_mod_sender']} ... Subject: {data['_mod_subject']} ... Date: {data['_mod_hold_date']} ... Reason: {data['_mod_reason']} ... """) ...

and then to delete the two messages:

 (venv) mailman@speedy:~/web/logs$ mailman shell -l sbarc-list@lists.netlojix.com Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the sbarc-list@lists.netlojix.com mailing list >>> requests = IListRequests(m) >>> first = True >>> for request in requests.held_requests: ...     key, data = requests.get_request(request.id) ...     if first: ...             first = False ...             print(m.list_id) ...     print(f'request id: {request.id}, deleting') ...     requests.delete_request(request.id) ... sbarc-list.lists.netlojix.com request id: 27, deleting request id: 28, deleting >>>

The result of this, however, was that the web interface still claimed I had two held messages.  I wasn't sure if there was some cron-type "sweeper" process I had to wait to auto-run or if I had to manually do something to create an update.  I then realized that the sample script had a ' >>> commit ()" ' at the end, which I'd missed.  So, I re-entered the script (but it didn't spit out that it deleted anything this time) and got an error when I did the commit.

Was this in the same mailman shell interaction or a new one? What was the error?

  So, I figured the messages HAD been delete and the commit wasn't really necessary.  When I went back to the web interface, the number of held messages was at 0 and I no longer get the original error when clicking on "Held messages", so my issue is solved. But this does leave me with some educational questions:

1. Did I affect something somehow with my "commit", or was there a job that cleaned things up (if so, what job)?

I'm not sure of the sequence of events, but I think you probably looked in Postorius before you quit mailman shell and then you quit mailman shell with ctrl-D which did an implicit commit.

2. Is the "commit" even necessary (I saw it in someone else's script in the thread)?

It is necessary to explicitly commit() or quit with ctrl-D to actually update the database.

3. Is there a way to save a script and then execute it via shell rather than typing it in every time (it appears from the examples that people are typing them in every time, which is super tedious; I didn't paste for fear of doing something damaging, but maybe that would work)?

Yes, you can save a script in the bin/ directory in your venv and run it with mailman shell -r. See mailman shell --details.

4. Are the request.id's of the messages dynamic/changing, or are they fixed for life?  I thought I'd delete based on exact request.id without looping but I was afraid maybe they were changing over time and that this wouldn't be a good idea.  Might be nice to know for the future.

The request.id is constant for that request object. The request object is fixed for a specific held message request. I.e., a specific held message's request object and its request.id and other attributes are fixed until that request is handled.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 6/26/23 4:19 PM, Ken Alker wrote:

What else is stored in /var/messages, if not archives?  I see 4711 messages in that directory (which I presume would be as many messages are in the archive database).  If there is a write-up on this, feel free to point me to it.  I found <https://docs.mailman3.org/projects/mailman/en/latest/build/lib/mailman/model...> but this isn't really answering my question.

These are probably artifacts from your old Debian package. There have been issues in the past with handled messages not being removed from the message store. See https://gitlab.com/mailman/mailman/-/issues/257

This was fixed in 3.3.5 and the task runner implemented in 3.3.5 removes orphaned messages from the message store, but if they are still referenced in pendings they won't be deleted.

I didn't think so, but I'll certainly buy that.  So, I'm assuming by this you are implying that there is no helper function that needed to run and the number of held messages should have dropped to zero immediately upon a refresh of the web page?

Correct, assuming your changes were committed.

So if I run the script, exist the shell, then come back in again, I can't rely on the request.id being the same?  I'd need to re-run the script and get a new request.id?

No. The request id for a given held message won't change.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 6/26/23 10:27 PM, Ken Alker wrote:

So is the only thing stored in var/messages/ the held messages, and nothing more?

Two kinds of messages are in the message store. They are messages which are held for moderator action and bounce DSN messages.

Held messages should be removed from the message store when they are handled by a moderator. If there are no held messages in Postorius for any list, any in the message store can be safely removed.

Bounce DSNs are only needed for a very short time. They are saved when a bounce is received and removed after a few days by the task runner, but they are only needed until they are attached to the admin notice which happens within minutes, and this was only implemented in Mailman 3.3.5. Prior to that bounce DSNs weren't saved.

Probably most if not all of the messages in your message store are not needed and could be deleted, but it's more complicated than that because there are also entries in various database tables that should be removed as well.

Is there an outline as to how to clean up the var/messages hierarchy?
(I assume one has to be careful not to remove current held messages and maybe old ones that have not been processed; or maybe "old ones" can all be assumed orphaned, and thus, removed w/o concern?)

You could try something like this in mailman shell

 >>> pendings = getUtility(IPendings)
 >>> messages = getUtility(IMessageStore)
 >>> mids = []
 >>> for mlist in getUtility(IListManager):
...     requests = IListRequests(mlist)
...     for request in requests.held_requests:
...         key, data = requests.get_request(request.id)
...         if data.get('_mod_message_id'):
...             mids.append(data.get('_mod_message_id'))
...
 >>> for key, data in pendings:
...     mid = data.get('_mod_message_id')
...     if data['type'] != 'probe' and mid and mid not in mids:
...         pendings.confirm(key, expunge=True)
...         messages.delete_message(mid)
...
 >>> commit()

This will find all the message ids from all held message requests, and 
then find those pendings for message ids not DSNs and not found in 
requests and delete the pending an the message store message.

-- 
Mark Sapiro <mark@msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan