Getting rid of held messages from the command line
Hello,
We're currently running some of Mailman software, as available via pip, namely:
- mailman 3.1.1 (pip3)
- postorius 1.1.2 (pip2)
- hyperkitty 1.1.4 (pip2)
- mailmanclient 3.1.1 (pip3)
So far so good, however, for some reasons, a couple of our lists have started to accumulate held messages that cannot be manipulated via the REST API. Regardless if we use the web interface or mailmanclient, it ends up with:
Mar 14 10:58:32 2018 (24287) REST request handler error:
Traceback (most recent call last):
File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run
self.result = application(self.environ, self.start_response)
File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 50, in wrapper
rtn = function(*args, **kws)
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/wsgiapp.py", line 214, in __call__
return super().__call__(environ, start_response)
File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__
responder(req, resp, **params)
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 167, in on_get
resource = self._make_collection(request)
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/helpers.py", line 159, in _make_collection
for resource in collection]
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/helpers.py", line 159, in <listcomp>
for resource in collection]
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 157, in _resource_as_dict
resource = self._make_resource(request.id)
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 70, in _make_resource
resource = super()._make_resource(request_id)
File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 39, in _make_resource
results = requests.get_request(request_id)
File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
return function(args[0], config.db.store, *args[1:], **kws)
File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request
result.data_hash, expunge=False)
File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
return function(args[0], config.db.store, *args[1:], **kws)
File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm
value = json.loads(keyvalue.value)
File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
Mar 14 10:58:32 2018 (24287) 127.0.0.1 - - "GET /3.1/lists/xx@xx.xx/held HTTP/1.1" 500 59
Is there a way to manipulate/see/remove these mails differently? Right now this prevent moderation of any held messages (note: some lists do not suffer from this and the held panel or mailmanclient works perfectly fine).
Thanks!
On 03/14/2018 03:04 AM, Aymeric Mansoux wrote:
So far so good, however, for some reasons, a couple of our lists have started to accumulate held messages that cannot be manipulated via the REST API. Regardless if we use the web interface or mailmanclient, it ends up with:
Mar 14 10:58:32 2018 (24287) REST request handler error: Traceback (most recent call last):
File "/usr/lib/python3.5/wsgiref/handlers.py", line 137, in run self.result = application(self.environ, self.start_response)
File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 50, in wrapper rtn = function(*args, **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/rest/wsgiapp.py", line 214, in __call__ return super().__call__(environ, start_response) File "/usr/local/lib/python3.5/dist-packages/falcon/api.py", line 244, in __call__ responder(req, resp, **params) File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 167, in on_get resource = self._make_collection(request) File "/usr/local/lib/python3.5/dist-packages/mailman/rest/helpers.py", line 159, in _make_collection for resource in collection] File "/usr/local/lib/python3.5/dist-packages/mailman/rest/helpers.py", line 159, in <listcomp> for resource in collection] File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 157, in _resource_as_dict resource = self._make_resource(request.id) File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 70, in _make_resource resource = super()._make_resource(request_id) File "/usr/local/lib/python3.5/dist-packages/mailman/rest/post_moderation.py", line 39, in _make_resource results = requests.get_request(request_id) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request result.data_hash, expunge=False) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm value = json.loads(keyvalue.value) File "/usr/lib/python3.5/json/__init__.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0) Mar 14 10:58:32 2018 (24287) 127.0.0.1 - - "GET /3.1/lists/xx@xx.xx/held HTTP/1.1" 500 59Is there a way to manipulate/see/remove these mails differently? Right now this prevent moderation of any held messages (note: some lists do not suffer from this and the held panel or mailmanclient works perfectly fine).
We've seen this issue before with a different underlying cause. See <https://gitlab.com/mailman/mailman/issues/256>.
There are a couple of problems here. The serious problem is if there is a held message which somehow throws an exception in processing in mailman/rest/post_moderation.py, that totally breaks the Postorius held messages view for that list. We need to fix that.
Also it would be good to see the problem message.
Here are some things you can do in 'mailman shell' (aka mailman withlist). It is not clear what might not work, but you can try
mailman shell -l list@domain
which will respond
Welcome to the GNU Mailman shell The variable 'm' is the list@domain mailing list
Then you should be able to do at the >>> prompts:
req_db = IListRequests(m) reqs = list(req_db.held_requests)
reqs is now a list of the held requests and you can do something like
for req in reqs: ... print('{}: {}'.format(req.id, req.request_type)) ...
or since req_db.held_requests is a generator
for req in rdb.held_requests: ... print('{}: {}'.format(req.id, req.request_type)) ...
either of which will produce a list like
1: RequestType.held_message
with the IDs and types of each request.
You can handle these requests with something like
from mailman.app.moderator import handle_message handle_message(m, 1, Action.defer)
where the second argument is the ID of the message and the action is one of Action.defer, Action.hold, Action.discard, Action.reject or Action.accept, the first two of which do nothing, but the others will do the indicated action.
You can see the actual messages like this
msg_db = getUtility(IMessageStore)
then
key = req[0].key msg_db.get_message_by_id(key)
which will return the actual held message that is in this case index 0 in the list of held_requests.
You may have to do
commit()
when done, although if you don't do abort(), just exiting with control-D should do a commit().
I hope you find this helpful, and if you do determine the content of the offending message(s), it will be helpful if you post that.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hello Mark,
Thanks for the quick reply!
Based on your instructions and reading a bit more in the manual, I made a couple of quick-and-dirty helping functions:
from mailman.app.moderator import handle_message list_manager = getUtility(IListManager) msg_db = getUtility(IMessageStore)
def get_lists(): for name in sorted(list_manager.names): print(name)
def get_held_messages(listname): m = list_manager.get(listname) req_db = IListRequests(m) reqs = list(req_db.held_requests) for req in reqs: print(req.key) for line in str(msg_db.get_message_by_id(req.key)).split("\n"): if any(x in line for x in ['From:', 'Subject:']): print(line) print('\n')
def export_held_messages(listname): m = list_manager.get(listname) req_db = IListRequests(m) reqs = list(req_db.held_requests) for req in reqs: with open('/tmp/' + listname, 'a+') as f: f.write(str(msg_db.get_message_by_id(req.key))) f.write('\n\n\n\n\n\n') f.close()
def discard_held_messages(listname): m = list_manager.get(listname) req_db = IListRequests(m) reqs = list(req_db.held_requests) for req in reqs: handle_message(m, req.id, Action.discard) commit()
However, while I was testing this happily on a list that was not affected with the JSON error, I finally tested discard_held_messages() on one of the problematic list, and I get the exact same problem as when trying to view the moderation queue API:
discard_held_messages_queue('xxx@xxx.xxx') Traceback (most recent call last): File "<console>", line 1, in <module> File "<console>", line 6, in discard_held_messages_queue File "/usr/local/lib/python3.5/dist-packages/mailman/app/moderator.py", line 97, in handle_message key, msgdata = requestdb.get_request(id) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request result.data_hash, expunge=False) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm value = json.loads(keyvalue.value) File "/usr/lib/python3.5/json/__init__.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
With that said, even though I cannot pass the discard action, I can export the queue without problem, so I will send you off-list a sample, because I am not sure what I should be looking for.
Finally, is there another way to get rid of these messages? Are there consequences if I would use message_store.delete_message(message_id) instead of handle_message(m, req.id, Action.discard)?
Thanks! a.
On 03/16/2018 05:01 PM, Aymeric Mansoux wrote:
However, while I was testing this happily on a list that was not affected with the JSON error, I finally tested discard_held_messages() on one of the problematic list, and I get the exact same problem as when trying to view the moderation queue API:
discard_held_messages_queue('xxx@xxx.xxx') Traceback (most recent call last): File "<console>", line 1, in <module> File "<console>", line 6, in discard_held_messages_queue File "/usr/local/lib/python3.5/dist-packages/mailman/app/moderator.py", line 97, in handle_message key, msgdata = requestdb.get_request(id) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request result.data_hash, expunge=False) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm value = json.loads(keyvalue.value) File "/usr/lib/python3.5/json/__init__.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
I don't know that much about core, but I assume that the value that is being decoded is from the database of Mailman so it might be helpful to look at the values that are in and maybe figure out why they aren't valid json.
I might be completely wrong though..
On 03/16/2018 09:01 AM, Aymeric Mansoux wrote:
However, while I was testing this happily on a list that was not affected with the JSON error, I finally tested discard_held_messages() on one of the problematic list, and I get the exact same problem as when trying to view the moderation queue API:
discard_held_messages_queue('xxx@xxx.xxx') Traceback (most recent call last): File "<console>", line 1, in <module> File "<console>", line 6, in discard_held_messages_queue File "/usr/local/lib/python3.5/dist-packages/mailman/app/moderator.py", line 97, in handle_message key, msgdata = requestdb.get_request(id) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request result.data_hash, expunge=False) File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm value = json.loads(keyvalue.value) File "/usr/lib/python3.5/json/__init__.py", line 319, in loads return _default_decoder.decode(s) File "/usr/lib/python3.5/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
With that said, even though I cannot pass the discard action, I can export the queue without problem, so I will send you off-list a sample, because I am not sure what I should be looking for.
Unfortunately, that didn't help. I was able to post all the messages in your sample to a test list and even with all the messages held, I could view them and discard them in postorius.
This may not be too surprising as looking at the traceback, the exception is thrown in trying to delete the pending confirmation from the pending database. The issue is when a message is held an entry is made in the pending database with a confirmation token that can be used for example to approve or discard the message by email.
There is an issue that affects MySQL databases only in that prior to the merge of <https://gitlab.com/mailman/mailman/merge_requests/333> values in the pendedkeyvalue table could be truncated (or exceptions thrown depending on MySQL settings). That appears to be the issue here.
The fix was only merged in January and is not in Mailman 3.1.1.
Finally, is there another way to get rid of these messages? Are there consequences if I would use message_store.delete_message(message_id) instead of handle_message(m, req.id, Action.discard)?
As I said, I think the issue is with the pendedkeyvalue table, not with the message itself. The table contains entries which are pickled versions of various things. A big issue is rule_misses. For one of your messages, the rule_misses list is
['dmarc-mitigation', 'no-senders', 'approved', 'emergency', 'loop', 'banned-address', 'member-moderation']
this gets pickled into
b'\x80\x03]q\x00(X\x10\x00\x00\x00dmarc-mitigationq\x01X\n\x00\x00\x00no-sendersq\x02X\x08\x00\x00\x00approvedq\x03X\t\x00\x00\x00emergencyq\x04X\x04\x00\x00\x00loopq\x05X\x0e\x00\x00\x00banned-addressq\x06X\x11\x00\x00\x00member-moderationq\x07e.'
which ultimately gets stored in the pendedkeyvalue table as a value for '_pck_rule_misses' which is
'"\u0080\u0003]q\u0000(X\u0010\u0000\u0000\u0000dmarc-mitigationq\u0001X\n\u0000\u0000\u0000no-sendersq\u0002X\b\u0000\u0000\u0000approvedq\u0003X\t\u0000\u0000\u0000emergencyq\u0004X\u0004\u0000\u0000\u0000loopq\u0005X\u000e\u0000\u0000\u0000banned-addressq\u0006X\u0011\u0000\u0000\u0000member-moderationq\u0007e."'
Which is longer than the VARCHAR(255) MySQL field for that value prior to <https://gitlab.com/mailman/mailman/merge_requests/333>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Aymeric Mansoux -
Mark Sapiro -
Simon Hanna