500 error on Held Messages page
Hello,
I searched the list but couldn't find anything specific to the error seeing with held messages for this specific list. See output below from the mailman.log. The problem doesn't seem to be related to the database listarchiver table. Please advise.
-- Dan Caballero Caltech Systems Administrator
MySQL [mailman3_prod]> select * from listarchiver where mailing_list_id='4899'; +------+-----------------+--------------+-------------+ | id | mailing_list_id | name | _is_enabled | +------+-----------------+--------------+-------------+ | 1872 | 4899 | mhonarc | 0 | | 1873 | 4899 | mail-archive | 0 | | 1874 | 4899 | prototype | 0 | | 1875 | 4899 | hyperkitty | 1 | +------+-----------------+--------------+-------------+ 4 rows in set (0.001 sec)
2022-09-20 14:44:17 [FALCON] [ERROR] GET /3.1/lists/jewishlife@caltech.edu/held?count=10&page=1 => Traceback (most recent call last): File "falcon/app.py", line 365, in falcon.app.App.__call__ File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/post_moderation.py", line 182, in on_get resource = self._make_collection(request) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/helpers.py", line 182, in _make_collection entries = [as_dict(resource) for resource in collection] File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/helpers.py", line 182, in <listcomp> entries = [as_dict(resource) for resource in collection] File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/post_moderation.py", line 173, in _resource_as_dict assert resource is not None, resource AssertionError: None
On 9/20/22 14:50, Dan Caballero wrote:
2022-09-20 14:44:17 [FALCON] [ERROR] GET /3.1/lists/jewishlife@caltech.edu/held?count=10&page=1 => Traceback (most recent call last): File "falcon/app.py", line 365, in falcon.app.App.__call__ File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/post_moderation.py", line 182, in on_get resource = self._make_collection(request) File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/helpers.py", line 182, in _make_collection entries = [as_dict(resource) for resource in collection] File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/helpers.py", line 182, in <listcomp> entries = [as_dict(resource) for resource in collection] File "/opt/mailmanve/lib/python3.9/site-packages/mailman/rest/post_moderation.py", line 173, in _resource_as_dict assert resource is not None, resource AssertionError: None
This is most likely a race condition between two browsers, one of which handles the held message while the other is getting the held messages.
Check Mailman's access.log entries with time stamps around 2022-09-20 14:44:17.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
The list owner/moderator mentioned seeing this shortly after she received a notification about a moderated message. It was a message she had sent just a few minutes before. The problem started last night. The count on the held messages shows "33" held messages. I'll see if I can just proceed with discarding the held messages via Mailman shell described here: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/Y...
I'm getting an unexpected error when I try to clear all the held messages. Any ideas what could be wrong here?
(mailmanve) mailman@11f52637274a:/var/www/mailman-www$ mailman shell -l jewishlife@caltech.edu Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the jewishlife@caltech.edu mailing list
from mailman.app.moderator import handle_message requestdb = IListRequests(m) for req in requestdb.held_requests: ... if req.request_type == RequestType.held_message: ... handle_message(m, req.id, Action.discard) ... Traceback (most recent call last): File "<console>", line 3, in <module> File "/opt/mailmanve/lib/python3.9/site-packages/mailman/app/moderator.py", line 112, in handle_message key, msgdata = requestdb.get_request(id) TypeError: cannot unpack non-iterable NoneType object
On 9/20/22 19:54, Dan Caballero wrote:
I'm getting an unexpected error when I try to clear all the held messages. Any ideas what could be wrong here?
> (mailmanve) mailman@11f52637274a:/var/www/mailman-www$ mailman shell -l jewishlife@caltech.edu
> Welcome to the GNU Mailman shell
> Use commit() to commit changes.
> Use abort() to discard changes since the last commit.
> Exit with ctrl+D does an implicit commit() but exit() does not.
> The variable 'm' is the jewishlife@caltech.edu mailing list
>>>> from mailman.app.moderator import handle_message
>>>> requestdb = IListRequests(m)
>>>> for req in requestdb.held_requests:
> ... if req.request_type == RequestType.held_message:
> ... handle_message(m, req.id, Action.discard)
> ...
> Traceback (most recent call last):
> File "<console>", line 3, in <module>
> File "/opt/mailmanve/lib/python3.9/site-packages/mailman/app/moderator.py", line 112, in handle_message
> key, msgdata = requestdb.get_request(id)
> TypeError: cannot unpack non-iterable NoneType object
There is a corrupt request with a request id that doesn't exist.
Try something like:
>>> from mailman.interfaces.
>>> requestdb = IListRequests(m)
>>> for req in requestdb.held_requests:
... if req.request_type == RequestType.held_message:
... if not requestdb.get_request(req.id):
... print(f'Bad request id: {req.id}, deleting')
... requestdb.delete_request(req.id)
...
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Looks like there's something missing.
What should I import from mailman.interfaces?
" >>> from mailman.interfaces."
== Dan
I figured out the correct import based on what was done previously. It worked.
">>> from mailman.app.moderator import handle_message
requestdb = IListRequests(m) for req in requestdb.held_requests: ... if req.request_type == RequestType.held_message: ... if not requestdb.get_request(req.id): ... print(f'Bad request id: {req.id}, deleting') ... requestdb.delete_request(req.id) ... Bad request id: 1694, deleting Bad request id: 2509, deleting Bad request id: 2727, deleting Bad request id: 3251, deleting Bad request id: 3265, deleting Bad request id: 3371, deleting Bad request id: 4844, deleting Bad request id: 6058, deleting Bad request id: 6201, deleting Bad request id: 6432, deleting Bad request id: 6502, deleting commit ()"
On 9/21/22 10:39, Dan Caballero wrote:
I figured out the correct import based on what was done previously. It worked.
Why do you think you need to import anything?
">>> from mailman.app.moderator import handle_message
handle_message is not referenced in the following and doesn't need to
be imported here.
>>>> requestdb = IListRequests(m)
>>>> for req in requestdb.held_requests:
> ... if req.request_type == RequestType.held_message:
> ... if not requestdb.get_request(req.id):
> ... print(f'Bad request id: {req.id}, deleting')
> ... requestdb.delete_request(req.id)
> ...
> Bad request id: 1694, deleting
> Bad request id: 2509, deleting
> Bad request id: 2727, deleting
> Bad request id: 3251, deleting
> Bad request id: 3265, deleting
> Bad request id: 3371, deleting
> Bad request id: 4844, deleting
> Bad request id: 6058, deleting
> Bad request id: 6201, deleting
> Bad request id: 6432, deleting
> Bad request id: 6502, deleting
>>>> commit ()"
I assume after doing this, you were able to access the held messages view for the list.
Note that the only names referenced in the above and not defined there
are IListRequests and RequestType, but these are among the many
names implicitly imported by mailman shell in interactive mode.
$ mailman shell
Welcome to the GNU Mailman shell
Use commit() to commit changes.
Use abort() to discard changes since the last commit.
Exit with ctrl+D does an implicit commit() but exit() does not.
>>> for name in dir():
... print(name)
...
ALL_TEMPLATES
ALT_TEMPLATE_NAMES
ALWAYS_REPLY
AcceptEvent
AcceptOwnerEvent
Action
AddressAlreadyLinkedError
AddressError
AddressNotLinkedError
AddressVerificationEvent
AlreadySubscribedError
ArchivePolicy
ArchiveRenderingMode
BadDomainSpecificationError
BounceContext
ChainEvent
ClobberDate
ConfigurationUpdatedEvent
ContinueProcessing
DMARCMitigateAction
DatabaseError
DeliveryMode
DeliveryStatus
DigestFrequency
DiscardEvent
DiscardMessage
DomainCreatedEvent
DomainCreatingEvent
DomainDeletedEvent
DomainDeletingEvent
DuplicateStyleError
EmailError
ExistingAddressError
FilterAction
FilterType
HoldEvent
IAPI
IAcceptableAlias
IAcceptableAliasSet
IAddress
IArchiver
IAutoResponseRecord
IAutoResponseSet
IBan
IBanManager
IBounceEvent
IBounceProcessor
ICLISubCommand
ICacheManager
IChain
IChainIterator
IChainLink
IConfiguration
IContentFilter
IDatabase
IDatabaseFactory
IDomain
IDomainManager
IEmailCommand
IEmailResults
IEmailValidator
IHandler
IHeaderMatch
IHeaderMatchList
ILanguage
ILanguageManager
IListArchiver
IListArchiverSet
IListManager
IListRequests
IMailTransportAgentAliases
IMailTransportAgentDelivery
IMailTransportAgentLifecycle
IMailingList
IMember
IMembershipManager
IMessage
IMessageStore
IMutableChain
IOneLastDigest
IPendable
IPended
IPendedKeyValue
IPendings
IPipeline
IPlugin
IPreferences
IRoster
IRule
IRunner
IStyle
IStyleManager
ISubscriptionManager
ISubscriptionService
ISwitchboard
ITemplate
ITemplateLoader
ITemplateManager
IUser
IUserManager
IWorkflowState
IWorkflowStateManager
InvalidConfigurationError
InvalidEmailAddressError
InvalidListNameError
LinkAction
ListAlreadyExistsError
ListCreatedEvent
ListCreatingEvent
ListDeletedEvent
ListDeletingEvent
MailmanError
MemberRole
MembershipChangeEvent
MembershipError
MembershipIsBannedError
MissingConfigurationFileError
MissingPreferredAddressError
MissingUserError
NewsgroupModeration
NoSuchListError
NotAMemberError
PasswordChangeEvent
PendType
Personalization
RejectEvent
RejectMessage
ReplyToMunging
RequestRecord
RequestType
Response
ResponseAction
RunnerCrashEvent
RunnerInterrupt
SomeRecipientsFailed
SubscriptionConfirmationNeededEvent
SubscriptionEvent
SubscriptionInvitationNeededEvent
SubscriptionMode
SubscriptionPendingError
SubscriptionPolicy
TokenOwner
TooManyMembersError
UnrecognizedBounceDisposition
UnsubscriptionConfirmationNeededEvent
UnsubscriptionEvent
UnverifiedAddressError
__builtins__
abort
commit
config
getUtility
m
>>>
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark,
I'm seeing this issue pop up this week with several lists. You previously mentioned two conflicting browsers potentially causing the issue. Can you elaborate?
In addition, is there a way for me to run this cleanup using mailman shell -r "somescript" so that I can check all the lists instead of one by one?
Thanks as always.
On 9/21/22 16:24, Dan Caballero wrote:
Mark,
I'm seeing this issue pop up this week with several lists. You previously mentioned two conflicting browsers potentially causing the issue. Can you elaborate?
That would only cause a transient issue which would not occur on a subsequent get of the held message view. I gather this is not the case here. I.e. once the error occurs on a get of the held message view, it is persistent.
In addition, is there a way for me to run this cleanup using mailman shell -r "somescript" so that I can check all the lists instead of one by one?
For a -r script, try
# In non-interactive scripts, we need the explicit imports.
from mailman.interfaces.listmanager import IListManager
from mailman.interfaces.requests import IListRequests, RequestType
from zope.component import getUtility
def somescript():
for mlist in getUtility(IListManager):
requestdb = IListRequests(mlist)
for req in requestdb.held_requests:
if req.request_type == RequestType.held_message:
if not requestdb.get_request(req.id):
print(f'List: {mlist.list_id}, Bad request id:
{req.id}, deleting')
requestdb.delete_request(req.id)
The real question here is what are these spurious requests and how are they occurring. If you do a query like
SELECT * FROM _request;
Are there entries with empty data_hash or entries with an id that is
not the value of any pended_id in the pendedkeyvalue table.
BTW, I understand why you thought an import was needed in the interactive script in my message at https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
- the
from mailman.interfaces.line was a mistake on my part and shouldn't have been there.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I just did a test run in my dev system and the script above removes nearly 2000 bad request ids.
Even after the cleanup using the script above the pendedkeyvalue table still has 648938 rows. Is this all just poor moderation from our owners/moderators not managing their lists?
Or is there some kind of condition where a process isn't completing?
Thanks again.
-- Dan
On 9/26/22 11:06, Dan Caballero wrote:
Even after the cleanup using the script above the pendedkeyvalue table still has 648938 rows. Is this all just poor moderation from our owners/moderators not managing their lists?
See https://gitlab.com/mailman/mailman/-/issues/853 and perhaps other issues referenced in that one.
What Mailman core version is this? 3.3.5 fixed a lot of this.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I have the same issue now. that's happen when I try to open held messages
This is what I have in my logs:
File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 167, in _resource_as_dict assert resource is not None, resource AssertionError: None
that is the default mailman3 debian package.
GNU Mailman 3.3.3 (Tom Sawyer)
things were working smoothly for a very long time. it started after disk array disaster. we installed the thing to another server, and restored the database from backups.
I've tried to use this magic:
The variable 'm' is the odoo_leads@qarea.org mailing list
from mailman.app.moderator import handle_message requestdb = IListRequests(m) for req in requestdb.held_requests: ... if req.request_type == RequestType.held_message: ... if not requestdb.get_request(req.id): ... print(f'Bad request id: {req.id}, deleting') ... requestdb.delete_request(req.id) ...
but no success. is there any way to fix this? the system is Debian 11.6 mailman installed from .deb package I am absolutely sure there wasn't any race condition, it's only me who accesses the server. I'd rather not install anything not from the repos, only if I will be sure, that I could not fix the issue the other way.
thank you.
On 3/30/23 07:36, Gregory Edigarov wrote:
I have the same issue now. that's happen when I try to open held messages
This is what I have in my logs:
File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 167, in _resource_as_dict assert resource is not None, resource AssertionError: None
What is the full traceback?
that is the default mailman3 debian package.
And your primary support resource should be Debian.
GNU Mailman 3.3.3 (Tom Sawyer)
things were working smoothly for a very long time. it started after disk array disaster. we installed the thing to another server, and restored the database from backups.
I've tried to use this magic:
> The variable 'm' is the odoo_leads@qarea.org mailing list
>>>> from mailman.app.moderator import handle_message
>>>> requestdb = IListRequests(m)
>>>> for req in requestdb.held_requests:
> ... if req.request_type == RequestType.held_message:
> ... if not requestdb.get_request(req.id):
> ... print(f'Bad request id: {req.id}, deleting')
> ... requestdb.delete_request(req.id)
> ...
>>>>
but no success. is there any way to fix this?
I think you're on the right track with the above mailman shell, but
evidently the issue is different.
You might try
>>> requestdb = IListRequests(m)
>>> for req in requestdb.of_type(RequestType.held_message):
... if not requestdb.get_request(req.id):
... print(f'Bad request id: {req.id}, deleting')
... requestdb.delete_request(req.id)
... else:
... print(f'Held request: {requestdb.get_request(req.id)}
...
>>>
To see what the requests look like.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Dan Caballero -
Gregory Edigarov -
Mark Sapiro