certain msgs not archived
Hi good folks,
I'm new to the group, so bear with me please. A decade or more with Mm2.1 but recently upgraded to Mm3 so no experiences with this latter.
Details: Debian 11.2 (stable), everything installed from deb packages, mariadb10.5, python3.9, mailman3.3.3, hyperkitty1.3.4, etc. everything stock Debian. A few low volume lists, one of them migrated from Mm2.1 but most started newly. Couple of glitches, resolved them most so far. but could not cope with this one yet so I must ask your advice.
On one of my maillists certain messages can't seem to make it to the archive. They spread right among list members but stuck in /var/lib/mailman3/archives/hyperkitty/spool and hyperkitty can not handle and archive them due to finding an incorrect string. This is a mailman-web.log excerpt below believed to be relevant but please just tell me if you need further details. I didn't want to flood with longer quotation for now but I can share full Django error email if needed.
12:18:19 [Q] INFO Enqueued 8697
ERROR 2022-02-16 12:18:19,310 5898 django.request Internal Server Error: /mailman3/hyperkitty/api/mailman/archive
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 226, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8C\\xB9\\x0D\\x0A...' for column mailman3web.hyperkitty_email.conten t at row 1")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py", line 81, in _decorator
return func(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py", line 127, in archive
add_to_list(mlist_fqdn, msg)
File "/usr/lib/python3/dist-packages/hyperkitty/lib/incoming.py", line 153, in add_to_list
email.save()
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 743, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 780, in save_base
updated = self._save_table(
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 873, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 910, in _do_insert
return manager._insert([self], fields=fields, return_id=update_pk,
File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 226, in query
_mysql.connection.query(self, query)
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8C\\xB9\\x0D\\x0A...' for column mailman3web.hyperkitty_email.content a
t row 1")
ERROR 2022-02-16 12:18:19,310 5898 django.request Internal Server Error: /mailman3/hyperkitty/api/mailman/archive
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 226, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8C\\xB9\\x0D\\x0A...' for column mailman3web.hyperkitty_email.conten t at row 1")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/http.py", line 40, in inner
return func(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py", line 81, in _decorator
return func(request, *args, **kwargs)
File "/usr/lib/python3/dist-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/hyperkitty/views/mailman.py", line 127, in archive
add_to_list(mlist_fqdn, msg)
File "/usr/lib/python3/dist-packages/hyperkitty/lib/incoming.py", line 153, in add_to_list
email.save()
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 743, in save
self.save_base(using=using, force_insert=force_insert,
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 780, in save_base
updated = self._save_table(
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 873, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 910, in _do_insert
return manager._insert([self], fields=fields, return_id=update_pk,
File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3/dist-packages/django/db/backends/mysql/base.py", line 71, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 209, in execute
res = self._query(query)
File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 315, in _query
db.query(q)
File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 226, in query
_mysql.connection.query(self, query)
Btw. I noticed this yesterday evening and there are 5 msgs stuck this way in spool so far.
Thanks for you anticipated help.
Sanyi
Meanwhile going ahead slowly. This resolved issue helped a lot:
https://gitlab.com/mailman/hyperkitty/-/issues/248
'charset': 'utf8mb4' into /etc/mailman3/mailman-web.py helped and now a message with utf8mb4 emoji char is archive properly.
But reading that particular msg in Hyperkitty spits out a new error:
Internal Server Error: /mailman3/hyperkitty/list/test1@dwatt.com/thread/OIHKKHNGNL6HF2SMFJZXSS6TABO2VK6Z/
AttributeError at /hyperkitty/list/test1@xxxxx.yyy/thread/OIHKKHNGNL6HF2SMFJZXSS6TABO2VK6Z/ 'NoneType' object has no attribute 'subject'
Request Method: GET Request URL: https://xxxxx.yyy/mailman3/hyperkitty/list/test1@xxxxx.yyy/thread/OIHKKHNGNL... Django Version: 2.2.25 Python Executable: /usr/bin/uwsgi-core Python Version: 3.9.2 Python Path: ['.', '', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.9/dist-packages'] Server time: Wed, 16 Feb 2022 21:27:14 +0100
Googling the mailman-users list I found another case where hyperkitty_thread table had a NULL starting_email_id and yesss! It has a NULL in starting_email_id really!
Now the billion$ question: what this particular msg with an utf8mb4 char has to do with starting_email_id??
Sanyi
Hmmm. Disregard my last msg about the new error. I can not reproduce and seems like the starting_email_id is no longer NULL there. I'm a lil confused now and will have to double check things.
Sanyi
After a day of testing thoroughly I'm happy as a pig in mud: archiving works correctly! utf8mb4 encoded characters no longer halt processing.
Probably also this config insufficiency caused hyperkitty_import fail a great number of Mm2.1 messages. So will get back to it and see if works right now and if I can also merge old Mm2.1 mbox messages with recent ones posted since I started Mm3. Something like saving recent mbox messages first, then deleting them, then importing old 2.1 mbox, then importing saved Mm3 mbox, then indexing, ... then scratching my head why all this does not work the way I thought.
Will sure raise more issues.
Sanyi
On 2/17/22 09:18, Sandor Dibuz wrote:
Probably also this config insufficiency caused hyperkitty_import fail a great number of Mm2.1 messages. So will get back to it and see if works right now and if I can also merge old Mm2.1 mbox messages with recent ones posted since I started Mm3. Something like saving recent mbox messages first, then deleting them, then importing old 2.1 mbox, then importing saved Mm3 mbox, then indexing, ... then scratching my head why all this does not work the way I thought.
If you rerun hyperkitty_import with the since= option set to a date earlier that the oldest message to import, it should just work. Messages which are already in the archive (by Message-ID:) will be ignored by the import process.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/17/22 19:16, Mark Sapiro wrote:
If you rerun hyperkitty_import with the since= option set to a date earlier that the oldest message to import, it should just work. Messages which are already in the archive (by Message-ID:) will be ignored by the import process.
Great! So valuable info. Thank you, Mark. "earlier that the oldest message" you sure mean earlier then the oldest of already archived. Or since= any date of the mbox messages to be imported. Will check the hyperkitty_import for since= option format.
Yet another question. Is it supposed to stop minutely/hourly/etc. mailman3-web cron jobs during import to avoid any collisions? Or even mailman3-web completely but probably it needs to be running. I only ask because my last import attempt on a test maillist failed with
[ERROR/MainProcess] Failed indexing 1 - 2 (retry 5/5): (1712, 'Index hyperkitty_attachment is corrupted') (pid 5841): (1712, 'Index hyperkitty_attachment is corrupted')
Optimizing cured the hyperkitty_attachment InnoDB index corruption problem but scared the sh.. out of me for a moment...
Thank you for your help, Mark.
Sanyi
On 2/17/22 10:49, Sandor Dibuz wrote:
Great! So valuable info. Thank you, Mark. "earlier that the oldest message" you sure mean earlier then the oldest of already archived. Or since= any date of the mbox messages to be imported. Will check the hyperkitty_import for since= option format.
That was a typo. I meant "earlier than". I tend to use since=1970-1-1.
It doesn't matter how early as long as it's earlier than the oldest
message to be imported.
Yet another question. Is it supposed to stop minutely/hourly/etc. mailman3-web cron jobs during import to avoid any collisions? Or even mailman3-web completely but probably it needs to be running. I only ask because my last import attempt on a test maillist failed with
[ERROR/MainProcess] Failed indexing 1 - 2 (retry 5/5): (1712, 'Index hyperkitty_attachment is corrupted') (pid 5841): (1712, 'Index hyperkitty_attachment is corrupted')
Optimizing cured the hyperkitty_attachment InnoDB index corruption problem but scared the sh.. out of me for a moment...
I don't know what may have caused the index corruption. I don't bother to stop any crons or services while hyperkitty_import is running and I've never had an issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark, thank you for the valuable help. Using option "since" made it way easier importing than saving/deleting/merging/goofing for hours.
So I managed to import old Mm2.1 mbox messages and also keep recent Mm3 ones for all my mailing lists. Had to cope with additional errors and eg. set higher SQL max_allowed_packet size due to big attachments, etc. but finally everything worked correctly. Yet I will start indexing but I left this for late night as it takes a long time on this modest hardware (Orange Pi+ 2E SBC).
Sanyi
participants (2)
-
Mark Sapiro -
Sandor Dibuz