The doubled entries issue
I've got the problem Multiple rows were found when one or none was required. I found the issue and a script here: https://gitlab.com/mailman/mailman/-/issues/657
I did su - mailman3 and ran:
python scriptname.py
and got:
Traceback (most recent call last): File "/opt/mailman3/fix-doubled-entries.py", line 21, in <module> listmgr = get_list_manager() ^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/fix-doubled-entries.py", line 6, in get_list_manager return getUtility(IListManager) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/zope/component/_api.py ", line 180, in getUtility raise ComponentLookupError(interface, name) zope.interface.interfaces.ComponentLookupError: (<InterfaceClass mailman.interfa ces.listmanager.IListManager>, '') (venv) 1[mailman3@dalen ~]$
Do I miss something? How can I fix the problem?
Thanks, Lars
I wrote:
I've got the problem Multiple rows were found when one or none was required.
I don't have much experience with databases. I did:
sudo -i -u postgres
psql mailman
select * from listarchiver where name = 'mhonarc';
and get:
id | mailing_list_id | name | _is_enabled ----+-----------------+---------+------------- 1 | 2 | mhonarc | f 9 | 4 | mhonarc | f 13 | 5 | mhonarc | f 17 | 6 | mhonarc | f 21 | 8 | mhonarc | f 25 | 9 | mhonarc | f 29 | 10 | mhonarc | f 33 | 7 | mhonarc | f 37 | 11 | mhonarc | f 41 | 12 | mhonarc | f 45 | 13 | mhonarc | f 49 | 14 | mhonarc | f 53 | 16 | mhonarc | f (13 rows)
mailman=#
So which lines should I remove, and how do I do that?
The issue occured after creating a new list and imported from an old 21 list.
Thanks
Lars
Lars Bjørndal writes:
select * from listarchiver where name = 'mhonarc';
and get:
id | mailing_list_id | name | _is_enabled ----+-----------------+---------+------------- 1 | 2 | mhonarc | f 9 | 4 | mhonarc | f 13 | 5 | mhonarc | f 17 | 6 | mhonarc | f 21 | 8 | mhonarc | f 25 | 9 | mhonarc | f 29 | 10 | mhonarc | f 33 | 7 | mhonarc | f 37 | 11 | mhonarc | f 41 | 12 | mhonarc | f 45 | 13 | mhonarc | f 49 | 14 | mhonarc | f 53 | 16 | mhonarc | f (13 rows)
mailman=#
So which lines should I remove, and how do I do that?
None of them? You have one entry per mailing list, which is as it should be.
Also, your preceding message doesn't complain about the database, it complains
raise ComponentLookupError(interface, name) zope.interface.interfaces.ComponentLookupError: (<InterfaceClass mailman.interfaces.listmanager.IListManager>, '')
That says that "python scriptname.py" need the IListManager interface but can't find it. You didn't provide enough information to explain why that might be, so I'm not going to guess.
Tell us more about the "duplicate entries" error, and if possible copy it exactly from where you found it originally, including any traceback.
Also tell us more about your installation. Is it a distro installation? If so, which one, what version of Mailman? Is it a venv installation? If so, did you activate the venv before invoking "python scriptname.py"? If you didn't, try that and run scriptname again.
Steve
On Sat, 11 Nov 2023, at 10:15 PM, Stephen J. Turnbull wrote:
Lars Bjørndal writes:
raise ComponentLookupError(interface, name) zope.interface.interfaces.ComponentLookupError: (<InterfaceClass mailman.interfaces.listmanager.IListManager>, '')
That says that "python scriptname.py" need the IListManager interface but can't find it. You didn't provide enough information to explain why that might be, so I'm not going to guess.
This is mostly because of trying to run script with python scriptname.py
but since it doesn’t initialize the mailman’s zope system.
What you probably want to do is run the script using mailman shell —run
command, which will ensure that it runs the script after initializing the
zope components (and other mailman config and things are initialized too).
-- thanks, Abhilash Raj (maxking)
[Abhilash]
On Sat, 11 Nov 2023, at 10:15 PM, Stephen J. Turnbull wrote:
Lars Bjørndal writes:
raise ComponentLookupError(interface, name) zope.interface.interfaces.ComponentLookupError: (<InterfaceClass mailman.interfaces.listmanager.IListManager>, '')
That says that "python scriptname.py" need the IListManager interface but can't find it. You didn't provide enough information to explain why that might be, so I'm not going to guess.
This is mostly because of trying to run script with
python scriptname.pybut since it doesn’t initialize the mailman’s zope system.What you probably want to do is run the script using
mailman shell —runcommand, which will ensure that it runs the script after initializing the zope components (and other mailman config and things are initialized too).
Thank you. The script I try to run, is found here: https://gitlab.com/mailman/mailman/-/issues/657
I named the script fix-doubled-entries.py. When I try
mailman shell --run ./fix-doubled-entries.py I get:
Traceback (most recent call last): File "/opt/mailman3/venv/bin/mailman", line 8, in <module> sys.exit(main()) ^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/commands/cli_withlist.py", line 282, in shell r = call_name(dotted_name, *run_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/utilities/modules.py", line 69, in call_name named_callable = find_name(dotted_name) ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/utilities/modules.py", line 52, in find_name module = import_module(module_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib64/python3.11/importlib/__init__.py", line 121, in import_module raise TypeError(msg.format(name)) TypeError: the 'package' argument is required to perform a relative import for './fix-doubled-entries'
How should I invoke the script?
Thanks,
Lars
On 11/11/23 23:56, Lars Bjørndal wrote:
Thank you. The script I try to run, is found here: https://gitlab.com/mailman/mailman/-/issues/657
I named the script fix-doubled-entries.py. When I try
mailman shell --run ./fix-doubled-entries.pyI get:
This is not the way to run a script with mailman shell. See
mailman shell --details
for info. However, that script cannot be run as is by mailman shell --run. It can be run by invoking mailman shell interactively and
copying and pasting it, or if your Mailman is installed in a venv it can
be run with the venv active by python ./fix-doubled-entries.py.
However, that script will not fix your issue because you don't have multiple mhonarc entries for a list. See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... and as it says, Tell us more about the "duplicate entries" error, and if possible copy it exactly from where you found it originally, including any traceback.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi!
The problem seesm to be solved. Anyway, here's the answer to questions:
On Sun, Nov 12, 2023 at 08:07:59AM -0800, Mark Sapiro wrote:
On 11/11/23 23:56, Lars Bjørndal wrote:
Thank you. The script I try to run, is found here: https://gitlab.com/mailman/mailman/-/issues/657
I named the script fix-doubled-entries.py. When I try
mailman shell --run ./fix-doubled-entries.pyI get:This is not the way to run a script with
mailman shell. Seemailman shell --detailsfor info. However, that script cannot be run as is by
mailman shell --run. It can be run by invokingmailman shellinteractively and copying and pasting it,
Yes, that worked, e.g. no error messages.
or if your Mailman is installed in a venv it can be run with the venv active by
python ./fix-doubled-entries.py.
I have mailman installed in venv, but got the following when trying that:
"python fix-doubled-entries.py" need the IListManager interface but can't find it.
However, that script will not fix your issue because you don't have multiple mhonarc entries for a list. See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... and as it says, Tell us more about the "duplicate entries" error, and if possible copy it exactly from where you found it originally, including any traceback.
Here's the entries in the mailman.log file:
[11/Nov/2023:09:18:41 +0100] "GET /3.1/addresses/lars@lamasti.net HTTP/1.1" 200 365 "-" "GNU Mailman REST client v3.3.5" [11/Nov/2023:09:18:42 +0100] "GET /3.1/lists/annonsering.hodr.no/roster/moderator HTTP/1.1" 200 90 "-" "GNU Mailman REST client v3.3.5" Nov 11 09:18:43 2023 (1709) HyperKitty archived message <m3sf5cd7ok.fsf@dalen.lamasti.net> to https://hodr.no/archives/list/annonsering@hodr.no/message/NL5LTYANYLKOF5N33T... [11/Nov/2023:09:18:44 +0100] "GET /3.1/users/lars@lamasti.net HTTP/1.1" 200 259 "-" "GNU Mailman REST client v3.3.5" Nov 11 09:18:47 2023 (1715) Uncaught runner exception: Multiple rows were found when one or none was required Nov 11 09:18:47 2023 (1715) Traceback (most recent call last): File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/runners/outgoing.py", line 111, in _dispose self._func(mlist, msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/mta/deliver.py", line 89, in deliver refused = agent.deliver(mlist, msg, msgdata) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/mta/base.py", line 167, in deliver callback(mlist, message_copy, msgdata_copy) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/mta/decorating.py", line 32, in decorate decorator.process(mlist, msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/handlers/decorate.py", line 250, in process process(mlist, msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/handlers/decorate.py", line 73, in process for archiver in IListArchiverSet(mlist).archivers: ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/zope/component/_api.py", line 167, in adapter_hook return sitemanager.queryAdapter(object, interface, name, default) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/zope/interface/registry.py", line 351, in queryAdapter return self.adapters.queryAdapter(object, interface, name, default) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/database/transaction.py", line 106, in wrapper return function(args[0], config.db.store, *args[1:], **kws) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/model/mailinglist.py", line 653, in __init__ ListArchiver.name == archiver_name).one_or_none() ^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2769, in one_or_none return self._iter().one_or_none() # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/sqlalchemy/engine/result.py", line 1813, in one_or_none return self._only_one_row( ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/sqlalchemy/engine/result.py", line 816, in _only_one_row raise exc.MultipleResultsFound( sqlalchemy.exc.MultipleResultsFound: Multiple rows were found when one or none was required Nov 11 09:18:47 2023 (1715) SHUNTING: 1699690727.810848+3b6f3baba5576f3d87d3ff392aba2715d4dda793 [11/Nov/2023:10:00:02 +0100] "GET /3.1/lists?count=10&page=1 HTTP/1.1" 200 4061 "-" "GNU Mailman REST client v3.3.5"
I don't know what fixed the problem, but when I did mailman unshunt the message went through.
Will mailman notify when there's shunted messages?
Lars
On 11/13/23 10:06, Lars Bjørndal wrote:
On Sun, Nov 12, 2023 at 08:07:59AM -0800, Mark Sapiro wrote:
for info. However, that script cannot be run as is by
mailman shell --run. It can be run by invokingmailman shellinteractively and copying and pasting it,Yes, that worked, e.g. no error messages.
or if your Mailman is installed in a venv it can be run with the venv active by
python ./fix-doubled-entries.py.I have mailman installed in venv, but got the following when trying that:
"python fix-doubled-entries.py" need the IListManager interface but can't find it.
I see that that's actually expected. In order to run that script with the Python in the venv, you'd need to add
from mailman.core import initialize
initialize.initialize()
to the script to properly set up the Zope components.
However, that script will not fix your issue because you don't have multiple mhonarc entries for a list. See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... and as it says, Tell us more about the "duplicate entries" error, and if possible copy it exactly from where you found it originally, including any traceback.
Here's the entries in the mailman.log file:
...
Nov 11 09:18:47 2023 (1715) Uncaught runner exception: Multiple rows were found when one or none was required Nov 11 09:18:47 2023 (1715) Traceback (most recent call last): File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/core/runner.py", line 179, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/core/runner.py", line 272, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/runners/outgoing.py", line 111, in _dispose self._func(mlist, msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/mta/deliver.py", line 89, in deliver refused = agent.deliver(mlist, msg, msgdata) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/mta/base.py", line 167, in deliver callback(mlist, message_copy, msgdata_copy) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/mta/decorating.py", line 32, in decorate decorator.process(mlist, msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/handlers/decorate.py", line 250, in process process(mlist, msg, msgdata) File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/handlers/decorate.py", line 73, in process for archiver in IListArchiverSet(mlist).archivers: ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/zope/component/_api.py", line 167, in adapter_hook return sitemanager.queryAdapter(object, interface, name, default) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/zope/interface/registry.py", line 351, in queryAdapter return self.adapters.queryAdapter(object, interface, name, default) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/database/transaction.py", line 106, in wrapper return function(args[0], config.db.store, *args[1:], **kws) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/mailman/model/mailinglist.py", line 653, in __init__ ListArchiver.name == archiver_name).one_or_none() ^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 2769, in one_or_none return self._iter().one_or_none() # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/sqlalchemy/engine/result.py", line 1813, in one_or_none return self._only_one_row( ^^^^^^^^^^^^^^^^^^^ File "/opt/mailman3/venv/lib64/python3.11/site-packages/sqlalchemy/engine/result.py", line 816, in _only_one_row raise exc.MultipleResultsFound( sqlalchemy.exc.MultipleResultsFound: Multiple rows were found when one or none was required Nov 11 09:18:47 2023 (1715) SHUNTING: 1699690727.810848+3b6f3baba5576f3d87d3ff392aba2715d4dda793
That does look like https://gitlab.com/mailman/mailman/-/issues/657, but your database query showed no duplicate mhonarc entries for a list. Possibly, it was a different archiver, or more likely considering
I don't know what fixed the problem, but when I did
mailman unshuntthe message went through.
it was already fixed when you queried the database.
Will mailman notify when there's shunted messages?
No. I run a daily cron that does ls -lAR /opt/mailman/mm/var/queue/
which lists the shunt queue and the others.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Nov 13, 2023 at 08:58:05PM -0800, Mark Sapiro wrote:
On 11/13/23 10:06, Lars Bjørndal wrote:
[...]
Will mailman notify when there's shunted messages?
No. I run a daily cron that does
ls -lAR /opt/mailman/mm/var/queue/which lists the shunt queue and the others.
Today, I find the following file in the shunt folder:
ls -l /opt/mailman3/mm/var/queue/shunt/
-rw-rw---- 1 mailman3 mailman3 25473 des. 21 11:11 1703153465.6663258+bfbef88d4b3433eb5978cb42852601faee9ab00c.pck.tmp
What's this file? Should I take action or just remove it?
Thanks, Lars
Lars Bjørndal writes:
Today, I find the following file in the shunt folder:
ls -l /opt/mailman3/mm/var/queue/shunt/-rw-rw---- 1 mailman3 mailman3 25473 des. 21 11:11 1703153465.6663258+bfbef88d4b3433eb5978cb42852601faee9ab00c.pck.tmp
What's this file? Should I take action or just remove it?
The .pck.tmp and .pck.bak files that are occasionally observed are there to ensure that Mailman can recover if the system goes down while processing a message. The shunt queue is a little different in that nothing is ever done automatically to messages in that queue, there needs to be operator intervention ("unshunt").
Generally you can remove those files if they aren't removed automatically, but if you're concerned you can use "mailman qfile" to examine the content. (You may need to rename to remove the extension .tmp or .bak.)
Steve
On Sat, Dec 23, 2023 at 09:24:36AM +0900, Stephen J. Turnbull wrote:
[...]
The .pck.tmp and .pck.bak files that are occasionally observed are there to ensure that Mailman can recover if the system goes down while processing a message. The shunt queue is a little different in that nothing is ever done automatically to messages in that queue, there needs to be operator intervention ("unshunt").
Generally you can remove those files if they aren't removed automatically, but if you're concerned you can use "mailman qfile" to examine the content. (You may need to rename to remove the extension .tmp or .bak.)
Thanks. It was a graylisted message that came through.
Lars
参加者 (4)
-
Abhilash Raj -
Lars Bjørndal -
Mark Sapiro -
Stephen J. Turnbull