"It's in Mailman's database in the 'message' table."

I can see the message table - is there a way to see where it is marked as "held"? All I'm seeing in that table is id, message_id, message_id_hash, and path.

I'm wanting to be able to query for held messages for a given mailing list, as there are currently over 98,000 entries in the message table.

-Darren

On Mon, Apr 23, 2018 at 3:17 PM, Mark Sapiro <mark@msapiro.net> wrote:

...

So it appears that there was a message sent to one of our mailing lists that was put into the held message queue. However, when attempting to go to the message to do the moderation, the web page returns and Error 500.

Looking in the mailman log, I found this:

Apr 23 14:21:02 2018 (17878) REST request handler error: Traceback (most recent call last): File "/opt/mailman3/conda/lib/python3.6/wsgiref/handlers.py", line 137, in run self.result = application(self.environ, self.start_response) File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/database/transac tion.py", line 50, in wrapper rtn = function(*args, **kws) File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/rest/wsgiapp.py" , line 214, in __call__ return super().__call__(environ, start_response) File "/opt/mailman3/conda/lib/python3.6/site-packages/falcon/api.py", line 227 , in __call__ responder(req, resp, **params) File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/rest/post_modera tion.py", line 167, in on_get resource = self._make_collection(request) File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/rest/helpers.py" , line 159, in _make_collection for resource in collection] File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/rest/helpers.py" , line 159, in <listcomp> for resource in collection] File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/rest/post_modera tion.py", line 157, in _resource_as_dict resource = self._make_resource(request.id) File "/opt/mailman3/conda/lib/python3.6/site-packages/ mailman/rest/post_modera tion.py", line 100, in _make_resource make_header(decode_header(resource['subject']))) File "/opt/mailman3/conda/lib/python3.6/email/header.py", line 174, in make_he ader h.append(s, charset) File "/opt/mailman3/conda/lib/python3.6/email/header.py", line 295, in append s = s.decode(input_charset, errors) UnicodeDecodeError: 'gb2312' codec can't decode byte 0x87 in position 2: illegal multibyte sequence

It LOOKS as if there might be a character in the email somewhere -

On 04/23/2018 01:31 PM, Darren Smith wrote: possibly

...

in the subject or sender fields? - that is somehow unexpected.

It appears that the Subject: header is RFC 2047 encoded and there is an encoded word with charset gb2312 that contains an invalid encoding.

The message is apparently defective, but this kind of defect shouldn't cause a 500 error.

I can't tell for sure from the traceback if this is Postorius or some other application accessing the REST api. Is it Postorius or something else?

...

I am attempting to find the held message - can someone point me where I can go to see the message so that I can see what might be causing the problem?

It's in Mailman's database in the 'message' table.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

---

Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

On 04/23/2018 11:29 PM, Darren Smith wrote:

"It's in Mailman's database in the 'message' table."

I can see the message table - is there a way to see where it is marked as "held"? All I'm seeing in that table is id, message_id, message_id_hash, and path.

I'm wanting to be able to query for held messages for a given mailing list, as there are currently over 98,000 entries in the message table. This should be possible using the mailman shell http://mailman.readthedocs.io/en/latest/src/mailman/model/docs/requests.html...

I haven't used the shell extensively and didn't read the whole docs, but it looks like it's the right place to look. If you can't do that using the shell, I'd say it's worth a bug report :-)

Well, using the mailman shell we were able to find the request that was causing the problem - it was apparently a spam message that had some chinese characters in it.

We used the shell to delete the held request...

And postorius is still giving me an error 500.

If I'm not mistaken the held messages are kept in .pck files - do you know if when postorius asks for held messages through the API, does mailman go to the database or to the .pck? I would actually suspect the latter, or removing the request should have fixed the problem.

I'm hoping this makes sense...

On Mon, Apr 23, 2018 at 3:39 PM, Darren Smith <silas.crutherton@gmail.com> wrote:

Very good, I'll give that a try and let you know!

On Mon, Apr 23, 2018 at 3:37 PM, Simon Hanna <simon@hannaweb.eu> wrote:

...

...

"It's in Mailman's database in the 'message' table."

I can see the message table - is there a way to see where it is marked as "held"? All I'm seeing in that table is id, message_id, message_id_hash, and path.

I'm wanting to be able to query for held messages for a given mailing

On 04/23/2018 11:29 PM, Darren Smith wrote: list,

...

as there are currently over 98,000 entries in the message table. This should be possible using the mailman shell http://mailman.readthedocs.io/en/latest/src/mailman/model/do cs/requests.html?highlight=held

I haven't used the shell extensively and didn't read the whole docs, but it looks like it's the right place to look. If you can't do that using the shell, I'd say it's worth a bug report :-)

---

Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

OK chalk another one up to a python newbie...

commit() was the magic sauce that we needed. The error 500 is no longer happening on that mailing list.

We have three different lists so far that have gotten into this state, and we are going to work on the other two to make sure that this all works.

FYI if this is fixed somehow in a later release (later than 3.1.0) we are working on migrating to the latest mailman code.

-Darren

On Mon, Apr 23, 2018 at 4:20 PM, Mark Sapiro <mark@msapiro.net> wrote:

On 04/23/2018 02:59 PM, Darren Smith wrote:

...

Well, using the mailman shell we were able to find the request that was causing the problem - it was apparently a spam message that had some chinese characters in it.

I am trying to duplicate this issue in a test environment. I can create a message with an invalid RFC 2047 encoded subject, i.e.,

Subject: =?gb2312?Q?=ff=fe=de=21=a2?=

and send it to a list where it gets held, but I don't get a 500 in postorius and I see the held message with a raw (not decoded) header.

I think there were some recent changes that affect this. I'll look some more.

...

We used the shell to delete the held request...

And postorius is still giving me an error 500.

If I'm not mistaken the held messages are kept in .pck files - do you know if when postorius asks for held messages through the API, does mailman go to the database or to the .pck? I would actually suspect the latter, or removing the request should have fixed the problem.

They were held in .pck files in Mailman 2.1, but not in Mailman 3. They should just be in the database and deleting them via mailman shell should delete them as long as you quit the shell or commit().

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

---

Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

On 04/23/2018 03:20 PM, Mark Sapiro wrote:

They were held in .pck files in Mailman 2.1, but not in Mailman 3. They should just be in the database and deleting them via mailman shell should delete them as long as you quit the shell or commit().

The messages themselves are in Mailman's var/messages directory. For a message with, e.g.,

Message-ID-Hash: YNLR4PVUVNFRKCCWSG7EHEKEVMUQMFTS

the message will be in

var/messages/YN/LR/YNLR4PVUVNFRKCCWSG7EHEKEVMUQMFTS

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan