On Thu, Dec 8, 2022 at 10:41 PM Mark Sapiro <mark@msapiro.net> wrote:
On 12/8/22 07:23, Odhiambo Washington wrote:
MM3-3.3.7, Hyperkitty-1.3.6
While importing, I encountered several of this error:
<CUT> MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1") The above exception was the direct cause of the following exception: ... File
"/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/backends/mysql/base.py",
That's a 4-byte utf-8 encoding for an emoji. MySQL and MariaDB have to be told to accept 4-byte encodings.
In the DATABASES['default'] definition, you need to set
'OPTIONS': {'charset': 'utf8mb4'}
I added that:
[database] class: mailman.database.mysql.MySQLDatabase url: mysql+pymysql://mailman_user:XXXXXXXXX@localhost /mailmansuite?charset=utf8mb4&use_unicode=1
Also, although it doesn't affect this, see
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/databa...
I read this too. Did not pick much from there.
You might also need to modify the tables, particularly hyperkitty_email with something like
ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4, MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci, MODIFY content CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
So I removed my list, dropped the database, and proceeded as follows:
create database mailmansuite CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci; ALTER TABLE hyperkitty_email DEFAULT CHARACTER SET utf8mb4; ALTER TABLE hyperkitty_email MODIFY subject varchar(512) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY content longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY sender_name varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; ALTER TABLE hyperkitty_email MODIFY in_reply_to varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
I did all those (although looks overzealous) just to be on the safe side. I have seen people putting emojis in their names.
After that, I did the mailman-web migrate, created the list, imported the config from MM2 and then proceeded to import the archives. However, the errors still are there, and several messages were not imported
- I have several "failed to import" warnings.
<CUT> django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x8A, ...' for column 'content' at row 1")
Message 253d9d44-cc3e-fed1-144b-34da88c52b39@emailplus.org failed to import, skipping *<================== several of these "failed to import" messages* Computing thread structure Synchronizing properties with Mailman 905 emails left to refresh, checked 0 765 emails left to refresh, checked 0 Warming up cache Traceback (most recent call last): File "/opt/mailman/mm/venv/bin/django-admin", line 8, in <module> sys.exit(execute_from_command_line()) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line utility.execute() File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 402, in run_from_argv self.execute(*args, **cmd_options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_import.py", line 411, in handle call_command("hyperkitty_warm_up_cache", list_address) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 198, in call_command return command.execute(*args, **defaults) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/core/management/base.py", line 448, in execute output = self.handle(*args, **options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 53, in handle self.warm_up_mlist(mlist, options) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/management/commands/hyperkitty_warm_up_cache.py", line 61, in warm_up_mlist for thread in mlist.recent_threads: File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 162, in recent_threads return self.cached_values"recent_threads" File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/common.py", line 58, in __call__ return self.get_or_set(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in get_or_set return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/hyperkitty/models/mailinglist.py", line 317, in <listcomp> return [Thread.objects.get(pk=pk) for pk in thread_ids] File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/mailman/mm/venv/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get raise self.model.DoesNotExist( hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist. (venv) [mailman@gw ~/mm]$
It therefore means I have several broken threads in the archives. I am still willing to try other solutions if they are suggested.
And what do I make of the final error - *hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist*. ??
-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)