On 03/16/2018 09:01 AM, Aymeric Mansoux wrote:
However, while I was testing this happily on a list that was not affected with the JSON error, I finally tested discard_held_messages() on one of the problematic list, and I get the exact same problem as when trying to view the moderation queue API:
discard_held_messages_queue('xxx@xxx.xxx') Traceback (most recent call last): File "<console>", line 1, in <module> File "<console>", line 6, in discard_held_messages_queue File "/usr/local/lib/python3.5/dist-packages/mailman/app/moderator.py", line 97, in handle_message key, msgdata = requestdb.get_request(id) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request result.data_hash, expunge=False) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm value = json.loads(keyvalue.value) File "/usr/lib/python3.5/json/__init__.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
With that said, even though I cannot pass the discard action, I can export the queue without problem, so I will send you off-list a sample, because I am not sure what I should be looking for.
Unfortunately, that didn't help. I was able to post all the messages in your sample to a test list and even with all the messages held, I could view them and discard them in postorius.
This may not be too surprising as looking at the traceback, the exception is thrown in trying to delete the pending confirmation from the pending database. The issue is when a message is held an entry is made in the pending database with a confirmation token that can be used for example to approve or discard the message by email.
There is an issue that affects MySQL databases only in that prior to the merge of <https://gitlab.com/mailman/mailman/merge_requests/333> values in the pendedkeyvalue table could be truncated (or exceptions thrown depending on MySQL settings). That appears to be the issue here.
The fix was only merged in January and is not in Mailman 3.1.1.
Finally, is there another way to get rid of these messages? Are there consequences if I would use message_store.delete_message(message_id) instead of handle_message(m, req.id, Action.discard)?
As I said, I think the issue is with the pendedkeyvalue table, not with the message itself. The table contains entries which are pickled versions of various things. A big issue is rule_misses. For one of your messages, the rule_misses list is
['dmarc-mitigation', 'no-senders', 'approved', 'emergency', 'loop', 'banned-address', 'member-moderation']
this gets pickled into
b'\x80\x03]q\x00(X\x10\x00\x00\x00dmarc-mitigationq\x01X\n\x00\x00\x00no-sendersq\x02X\x08\x00\x00\x00approvedq\x03X\t\x00\x00\x00emergencyq\x04X\x04\x00\x00\x00loopq\x05X\x0e\x00\x00\x00banned-addressq\x06X\x11\x00\x00\x00member-moderationq\x07e.'
which ultimately gets stored in the pendedkeyvalue table as a value for '_pck_rule_misses' which is
'"\u0080\u0003]q\u0000(X\u0010\u0000\u0000\u0000dmarc-mitigationq\u0001X\n\u0000\u0000\u0000no-sendersq\u0002X\b\u0000\u0000\u0000approvedq\u0003X\t\u0000\u0000\u0000emergencyq\u0004X\u0004\u0000\u0000\u0000loopq\u0005X\u000e\u0000\u0000\u0000banned-addressq\u0006X\u0011\u0000\u0000\u0000member-moderationq\u0007e."'
Which is longer than the VARCHAR(255) MySQL field for that value prior to <https://gitlab.com/mailman/mailman/merge_requests/333>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan