MM3 shunting mails
I'm running a Mailman 3.1.0 installation on a CentOS 7 machine. Using RPMs from https://repos.fedorapeople.org/repos/abompard/hyperkitty/el-7/x86_64/
I do have problems with some lists occasionally going into some kind of shunt mode, where alla mails sent to them gets shunted. In the Mailman log I get entries for each shunted email:
Mar 17 11:54:11 2017 (6217) Uncaught runner exception: Multiple rows were found for one_or_none() Mar 17 11:54:11 2017 (6217) Traceback (most recent call last): File "/usr/lib/python3.4/site-packages/mailman/core/runner.py", line 171, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3.4/site-packages/mailman/core/runner.py", line 264, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3.4/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3.4/site-packages/mailman/core/pipelines.py", line 53, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3.4/site-packages/mailman/handlers/rfc_2369.py", line 121, in process process(mlist, msg, msgdata) File "/usr/lib/python3.4/site-packages/mailman/handlers/rfc_2369.py", line 70, in process archiver_set = IListArchiverSet(mlist) File "/usr/lib/python3.4/site-packages/zope/component/hookable.py", line 33, in __call__ return self.__implementation(*args, **kw) File "/usr/lib/python3.4/site-packages/zope/component/_api.py", line 156, in adapter_hook return sitemanager.queryAdapter(object, interface, name, default) File "/usr/lib64/python3.4/site-packages/zope/interface/registry.py", line 348, in queryAdapter return self.adapters.queryAdapter(object, interface, name, default) File "/usr/lib/python3.4/site-packages/mailman/database/transaction.py", line 85, in wrapper return function(args[0], config.db.store, *args[1:], **kws) File "/usr/lib/python3.4/site-packages/mailman/model/mailinglist.py", line 607, in __init__ ListArchiver.name == archiver_name).one_or_none() File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2733, in one_or_none "Multiple rows were found for one_or_none()") sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none() Mar 17 11:54:11 2017 (6217) SHUNTING: 1489748051.1949184+7a59df19c81493cb419306d620376efc0e381819
And so far my Google-fu hasn't been good enough to give me any useful answers to why this happens. If I dump one of the pickles using qfile I cannot see any complaints.
Most of the times, if I remove and recreate a failing list and run unshunt the shunted emails get processed and sent.
Anyone seen something similar?
Regards,
Anders Nilsson
-- Anders Nilsson, Ph.D. andersn@control.lth.se Dep. of Automatic Control, Lund University Phone: +46 (0)46 222 87 94, Fax: +46 (0)46 13 81 18 Mob: +46 (0)703 22 57 67
On 03/17/2017 03:59 AM, Anders Nilsson wrote:
I do have problems with some lists occasionally going into some kind of shunt mode, where alla mails sent to them gets shunted. In the Mailman log I get entries for each shunted email:
Mar 17 11:54:11 2017 (6217) Uncaught runner exception: Multiple rows were found for one_or_none()
...
File "/usr/lib/python3.4/site-packages/mailman/handlers/rfc_2369.py", line 70, in process archiver_set = IListArchiverSet(mlist)
mailman/handlers/rfc_2369.py is adding the List-* headers. Here it is looking up all the archivers that are enabled for this list.
...
File "/usr/lib/python3.4/site-packages/mailman/model/mailinglist.py", line 607, in __init__ ListArchiver.name == archiver_name).one_or_none()
and here we are in code that is trying to see if there are any system enabled archivers that are not already associated with this mailing list and if so, add them to the archivers enabled for this list.
The above is the last line of
exists = store.query(ListArchiver).filter(
ListArchiver.mailing_list == mailing_list,
ListArchiver.name == archiver_name).one_or_none()which is querying the archivers enabled for this list for the name of a system enabled archiver and expects to find at most one entry
File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2733, in one_or_none "Multiple rows were found for one_or_none()")
but finds more than one.
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none() Mar 17 11:54:11 2017 (6217) SHUNTING: 1489748051.1949184+7a59df19c81493cb419306d620376efc0e381819
So the cause is there appear to be multiple database entries for this list and some archiver name, but I don't know how that can happen.
I have not seen this error.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mar 17, 2017, at 11:22 AM, Mark Sapiro wrote:
The above is the last line of
exists = store.query(ListArchiver).filter( ListArchiver.mailing_list == mailing_list, ListArchiver.name == archiver_name).one_or_none()which is querying the archivers enabled for this list for the name of a system enabled archiver and expects to find at most one entry
File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2733, in one_or_none "Multiple rows were found for one_or_none()")
but finds more than one.
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none() Mar 17 11:54:11 2017 (6217) SHUNTING: 1489748051.1949184+7a59df19c81493cb419306d620376efc0e381819
So the cause is there appear to be multiple database entries for this list and some archiver name, but I don't know how that can happen.
I don't know either, especially since ListArchiverSet.__init__() has this:
# Add any system enabled archivers which aren't already associated
# with the mailing list.
for archiver_name in system_archivers:
exists = store.query(ListArchiver).filter(
ListArchiver.mailing_list == mailing_list,
ListArchiver.name == archiver_name).one_or_none()
if exists is None:
store.add(ListArchiver(mailing_list, archiver_name,
system_archivers[archiver_name]))It runs the exact same query, and only adds the archiver to the store if it doesn't already exist.
What does the following give you?
% mailman shell Welcome to the GNU Mailman shell
# You may have to use getUtility(...).get() to find the offending list m = list(getUtility(IListManager).mailing_lists)[0] m
from mailman.model.mailinglist import ListArchiver q = config.db.store.query(ListArchiver).filter( ... ListArchiver.mailing_list == m) list(a.name for a in q) ['mhonarc', 'prototype', 'mail-archive']
Cheers, -Barry
Sorry for necro'ing this thread, but I have exactly this issue on mailman 3.3.0. My instance has about 500 lists, and some of them seem to get corrupted in this way every once in a while.
I wrote a script which removes the duplicate entries, but this doesn't stop the issue from reappearing. Any idea on how to find out what is happening here?
The script I wrote: `from mailman.interfaces.listmanager import IListManager from zope.component import getUtility from mailman.config import config from mailman.model.mailinglist import ListArchiver
def get_list_manager(): return getUtility(IListManager)
def get_mhonarc_archivers(mlist, thedb): return list(thedb.query(ListArchiver).filter(ListArchiver.mailing_list == mlist, ListArchiver.name == 'mhonarc'))
def remove_last(archivers, mlist, thedb): lastid = archivers[1].id thedb.query(ListArchiver).filter(ListArchiver.mailing_list == mlist, ListArchiver.name == 'mhonarc', ListArchiver.id == lastid).delete()
def do_the_thing(mlists, thedb): do_commit = False for mlist in mlists: mhonarcs = get_mhonarc_archivers(mlist, thedb) if len(mhonarcs) > 1: print("{} has {}".format(mlist.fqdn_listname, len(mhonarcs))) remove_last(mhonarcs, mlist, thedb) do_commit = True return do_commit
listmgr = get_list_manager() mlists = list(listmgr.mailing_lists) store = config.db.store print() do_commit = do_the_thing(mlists, store) if do_commit: store.commit()`
In the script I am filtering for 'mhonarc' ListArchivers, since the duplicate ListArchivers always seem to be mhonarc archivers.
If this is not the appropriate place, please let me know - I would like to get to the bottom of this :).
Kind regards, Ike
On 12/5/19 1:40 PM, ikemulder@hotmail.com wrote:
In the script I am filtering for 'mhonarc' ListArchivers, since the duplicate ListArchivers always seem to be mhonarc archivers.
If this is not the appropriate place, please let me know - I would like to get to the bottom of this :).
I see you've also filed https://gitlab.com/mailman/mailman/issues/657. Thank you for that.
We are as puzzled as you are by this. It would help if you could detect this duplication when it happens and try to get more information. The problem is I can't see how to do that. I would start by adding some logging to the patch_put method of the ListArchivers class in mailman/rest/lists.py to log the list and the request. Then when you see the duplicate, you can check this log to see if anything suspicious happened just prior to that?
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (4)
-
Anders Nilsson -
Barry Warsaw -
ikemulder＠hotmail.com -
Mark Sapiro