How to fix lists with characters errors?
Hi,
I just found several lists have the issue like this regarding charset:
(venv) mailman@new-mailman3 ~ (v3.1.1)> mailman members listas-charset-issue@mailman.domain.com anders@domain.com beth@domain.com charlie@domain.com david@domain.com ellen@domain.com Traceback (most recent call last): File "/local/mailman/venv/bin/mailman", line 33, in <module> sys.exit(load_entry_point('mailman==3.3.9', 'console_scripts', 'mailman')()) File "/local/mailman/venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/local/mailman/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/local/mailman/venv/lib/python3.10/site-packages/mailman/bin/mailman.py", line 69, in invoke return super().invoke(ctx) File "/local/mailman/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/local/mailman/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/local/mailman/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/local/mailman/venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/local/mailman/venv/lib/python3.10/site-packages/mailman/commands/cli_members.py", line 222, in members display_members(ctx, mlist, role, regular, File "/local/mailman/venv/lib/python3.10/site-packages/mailman/commands/cli_members.py", line 109, in display_members print(f'{address.display_name} <{address.original_email}>', UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in position 9: ordinal not in range(256)
Since this service is running since time of python2, I guess the old lists handled accented characters was to save as latin-1. But how can I fix this now?
Note: apparently the list remains working just fine. It is just the command line failing. And perhaps some other side effect.
Best Regards, Helio Loureiro https://helio.loureiro.eng.br https://github.com/helioloureiro https://mastodon.social/@helioloureiro
On 3/22/24 05:37, Helio Loureiro wrote:
Hi,
I just found several lists have the issue like this regarding charset:
(venv) mailman@new-mailman3 ~ (v3.1.1)> mailman members ... Traceback (most recent call last): ... line 109, in display_members print(f'{address.display_name} <{address.original_email}>', UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in position 9: ordinal not in range(256)
There is something in your python that has set the encoding for stdout to latin1 instead of the default utf-8. E.g.
$ python3.10
Python 3.10.9 (main, Dec 11 2022, 12:59:24) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> name = 'abc\u0142def'
>>> print (f'{name}')
abcłdef
>>> import sys
>>> sys.stdout.reconfigure(encoding='latin1')
>>> print (f'{name}')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in
position 3: ordinal not in range(256)
>>>
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi, it is python 3.10. Problem is... some mailing lists were created long ago, with python2 something. Most probably 2.5. And the tables on mysql were set to latin1.
That I know. Question now is how to fix this. Any suggestion? What is the table? I probably can run some script for these specific lists that have the failure, import data as latim-1, and update back to the table as utf-8.
Best Regards, Helio Loureiro https://helio.loureiro.eng.br https://github.com/helioloureiro https://mastodon.social/@helioloureiro
On Fri, 22 Mar 2024 at 20:53, Mark Sapiro <mark@msapiro.net> wrote:
On 3/22/24 05:37, Helio Loureiro wrote:
Hi,
I just found several lists have the issue like this regarding charset:
(venv) mailman@new-mailman3 ~ (v3.1.1)> mailman members ... Traceback (most recent call last): ... line 109, in display_members print(f'{address.display_name} <{address.original_email}>', UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in position 9: ordinal not in range(256)
There is something in your python that has set the encoding for stdout to latin1 instead of the default utf-8. E.g.
$ python3.10 Python 3.10.9 (main, Dec 11 2022, 12:59:24) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> name = 'abc\u0142def' >>> print (f'{name}') abcłdef >>> import sys >>> sys.stdout.reconfigure(encoding='latin1') >>> print (f'{name}') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'latin-1' codec can't encode character '\u0142' in position 3: ordinal not in range(256) >>>-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to helio@loureiro.eng.br
On 3/23/24 11:56, Helio Loureiro wrote:
Hi, it is python 3.10. Problem is... some mailing lists were created long ago, with python2 something. Most probably 2.5. And the tables on mysql were set to latin1.
This has nothing to do with the data or the charsets/collation in the database. The issue is the locale setting in your python is causing stdout output from the `mailman members` command to be encoded as `latin1` and at least some user's display name contains a `\u0142` (i.e. `ł` character.
That I know. Question now is how to fix this. Any suggestion? What is the table? I probably can run some script for these specific lists that have the failure, import data as latim-1, and update back to the table as utf-8.
That's not the issue. The issue is the Python in your venv has sys.stdout.encoding set to latin1, probably for locale reasons. You ca apply this patch ``` --- a/src/mailman/commands/cli_members.py +++ b/src/mailman/commands/cli_members.py @@ -17,6 +17,7 @@ """The 'members' subcommand.""" +import sys import click from mailman.core.i18n import _ @@ -30,6 +31,9 @@ from zope.component import getUtility from zope.interface import implementer +sys.stdout.reconfigure(encoding='utf-8') + + def display_members(ctx, mlist, role, regular, digest, nomail, outfp, email_only, count_only): # Which type of digest recipients should we display? ``` to mailman/commands/cli_members.py to fix the issue. -- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi, Thanks a lot! Nice catch. It was indeed loading as latin-1. Not sure why, but that's a matter to discuss on the Ubuntu mailing list. Best Regards, Helio Loureiro https://helio.loureiro.eng.br https://github.com/helioloureiro https://mastodon.social/@helioloureiro On Sun, 24 Mar 2024 at 01:07, Mark Sapiro <mark@msapiro.net> wrote:
On 3/23/24 11:56, Helio Loureiro wrote:
Hi, it is python 3.10. Problem is... some mailing lists were created long ago, with python2 something. Most probably 2.5. And the tables on mysql were set to latin1.
This has nothing to do with the data or the charsets/collation in the database. The issue is the locale setting in your python is causing stdout output from the `mailman members` command to be encoded as `latin1` and at least some user's display name contains a `\u0142` (i.e. `ł` character.
That I know. Question now is how to fix this. Any suggestion? What is the table? I probably can run some script for these specific lists that have the failure, import data as latim-1, and update back to the table as utf-8.
That's not the issue. The issue is the Python in your venv has sys.stdout.encoding set to latin1, probably for locale reasons. You ca apply this patch ``` --- a/src/mailman/commands/cli_members.py +++ b/src/mailman/commands/cli_members.py @@ -17,6 +17,7 @@
"""The 'members' subcommand."""
+import sys import click
from mailman.core.i18n import _ @@ -30,6 +31,9 @@ from zope.component import getUtility from zope.interface import implementer
+sys.stdout.reconfigure(encoding='utf-8') + + def display_members(ctx, mlist, role, regular, digest, nomail, outfp, email_only, count_only): # Which type of digest recipients should we display? ``` to mailman/commands/cli_members.py to fix the issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
_______________________________________________ Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/ Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...
This message sent to helio@loureiro.eng.br
participants (2)
-
Helio Loureiro -
Mark Sapiro