Localization of auto-responses, question marks in welcome message
Hi,
I know there are already some posts on this subject. From reading the following seems to be the state of the affair:
the list's language settings determines which localized template to use
the list-_servers_ default setting determines which language to use for the subject (so the subject's language is a "global" server configuration setting).
Still, in my Ubuntu Impish Mailman 3.3.3 installation the welcome message is sent with charset us-ascii and contains question marks for any "special" -- i.e. non-ascii -- character.
Can this be changed s.t. the welcome message is correctly displayed?
Kind thanks
Claus
-- Claus-Justus Heine himself@claus-justus-heine.de http://www.claus-justus-heine.de/
Schatzmeister der Camerata Academica Freiburg e.V. --- www.cafev.de
Mmmh. So is this "won't fix"? No one?
The encoding of the welcome-message is wrong, regardless of the localization settings. The list server's language is (in this case) German, the list's default language is German, everything is then localized correctly (also the subject) with the very notable exception of the welcome-message, which seems to be hard-coded to us-ascii, every "special" character is replaced by a question mark.
As everything else works I assume that this is a bug in mailman v3.
The only work-around that I see here (besides of digging into the source code) is to manually replace "special" characters by localized replacements (e.g. ä by ae etc).
Is there any other work-around?
Kind thanks
Claus
Am 07.04.22 um 19:24 schrieb Claus-Justus Heine via Mailman-users:
Hi,
I know there are already some posts on this subject. From reading the following seems to be the state of the affair:
the list's language settings determines which localized template to use
the list-_servers_ default setting determines which language to use for the subject (so the subject's language is a "global" server configuration setting).
Still, in my Ubuntu Impish Mailman 3.3.3 installation the welcome message is sent with charset us-ascii and contains question marks for any "special" -- i.e. non-ascii -- character.
Can this be changed s.t. the welcome message is correctly displayed?
Kind thanks
Claus
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- Claus-Justus Heine himself@claus-justus-heine.de http://www.claus-justus-heine.de/
Schatzmeister der Camerata Academica Freiburg e.V. --- www.cafev.de
Claus-Justus Heine via Mailman-users writes:
Mmmh. So is this "won't fix"? No one?
Encoding issues are very difficult to diagnose from "it's wrong" without access to the actual server. However, Mailman 3 is designed to take advantage of Python's "universally encoded text" string representation (it happens to be wide-char Unicode, but that's not important to this issue). This means that we have to explicitly specify encoding when displaying text. On a monolingual site, if only one text is incorrectly encoded in the UI, almost always the template is the problem. Bugs in Mailman are of course always possible, but the current design dramatically improves the situation vs. Mailman 2.
The encoding of the welcome-message is wrong, regardless of the localization settings.
Are you using the default welcome message, or is it a custom message you added yourself? Either way, check that the template file in templates/de/list:user:notice:welcome.txt is encoded in UTF-8 (2 bytes per non-ASCII character), not in a Latin-1-ish encoding (1 byte per non-ASCII character) as was often the case for German in software until fairly recently.
Note that if you have a list-specific template, this will be stored in the RDBMS, not as a template in the file system.
As everything else works I assume that this is a bug in mailman v3.
Assume what you like, but it's probably easier and *much* quicker to fix if it's a user error. ;-) Another possibility, if you configured the template through the web, is an encoding error by the browser.
Is there any other work-around?
I think it's most likely that your welcome.txt template somehow is in a Latin-1 variant encoding rather than UTF-8. Figure out which encoding and reencode to UTF-8 using iconv or similar. If it's not that, I don't have any good ideas because you say other stuff is working. There's no reason why it would fail only for that template.
Are you sure other templates are working correctly?
Steve
Kind thanks for the response. I have to apologize: my "test user" had its preferred language set to en_US, so that was the reason for the question marks. So this was indeed a user-error (i.e. my fault).
Claus
Am 24.04.22 um 15:50 schrieb Stephen J. Turnbull:
Claus-Justus Heine via Mailman-users writes:
Mmmh. So is this "won't fix"? No one?
Encoding issues are very difficult to diagnose from "it's wrong" without access to the actual server. However, Mailman 3 is designed to take advantage of Python's "universally encoded text" string representation (it happens to be wide-char Unicode, but that's not important to this issue). This means that we have to explicitly specify encoding when displaying text. On a monolingual site, if only one text is incorrectly encoded in the UI, almost always the template is the problem. Bugs in Mailman are of course always possible, but the current design dramatically improves the situation vs. Mailman 2.
The encoding of the welcome-message is wrong, regardless of the localization settings.
Are you using the default welcome message, or is it a custom message you added yourself? Either way, check that the template file in templates/de/list:user:notice:welcome.txt is encoded in UTF-8 (2 bytes per non-ASCII character), not in a Latin-1-ish encoding (1 byte per non-ASCII character) as was often the case for German in software until fairly recently.
Note that if you have a list-specific template, this will be stored in the RDBMS, not as a template in the file system.
As everything else works I assume that this is a bug in mailman v3.
Assume what you like, but it's probably easier and *much* quicker to fix if it's a user error. ;-) Another possibility, if you configured the template through the web, is an encoding error by the browser.
Is there any other work-around?
I think it's most likely that your welcome.txt template somehow is in a Latin-1 variant encoding rather than UTF-8. Figure out which encoding and reencode to UTF-8 using iconv or similar. If it's not that, I don't have any good ideas because you say other stuff is working. There's no reason why it would fail only for that template.
Are you sure other templates are working correctly?
Steve
-- Claus-Justus Heine himself@claus-justus-heine.de http://www.claus-justus-heine.de/
Schatzmeister der Camerata Academica Freiburg e.V. --- www.cafev.de
Claus-Justus Heine via Mailman-users writes:
Kind thanks for the response. I have to apologize: my "test user" had its preferred language set to en_US, so that was the reason for the question marks. So this was indeed a user-error (i.e. my fault).
But we'd like to have that user get the English version of the message. One of two things seems reasonable:
You have customized the German template, but not the English template. In that case, I think it makes sense for Mailman to assume the German template has important information that the English does not, and send the German template with its appropriate encoding (UTF-8, nowadays). The fact that en_US can be encoded in ASCII should not affect the encoding of the de template, ever.[1]
You have not customized the German template. In that case the user should get the default English template.
How do those sound to you, as goals for Mailman behavoir?
So ISTM this is a Mailman issue, if Mailman is sending out the German message with question marks substituted for non-ASCII characters, or if Mailman is sending out the German message with US-ASCII as the Content-Type charset parameter. The only way I can see this as a user (admin) issue is if you overwrote the English template with German (and even then, it's nuts for Mailman not to use UTF-8 for English what with directed quotes, emoji, and all the other enhancements to the character repertoire that US English speakers use).
Can you tell me a little more about whether you configured any custom templates, and if so, how? That is, did you use the Postorius text field or overwrite files directly?
Steve
Footnotes: [1] Note to self: perhaps the admin UI can check the list of supported languages and warn if some but not all are customized.
Am 25.04.22 um 08:41 schrieb Stephen J. Turnbull:
Claus-Justus Heine via Mailman-users writes:
Kind thanks for the response. I have to apologize: my "test user" had its preferred language set to en_US, so that was the reason for the question marks. So this was indeed a user-error (i.e. my fault).
But we'd like to have that user get the English version of the message. One of two things seems reasonable:
- You have customized the German template, but not the English template. In that case, I think it makes sense for Mailman to assume the German template has important information that the English does not, and send the German template with its appropriate encoding (UTF-8, nowadays). The fact that en_US can be encoded in ASCII should not affect the encoding of the de template, ever.[1]
I have installed custom templates using the REST API and pointing MM3 to some download URIs. So MM3 has custom templates for particular lists, but I did not yet install different templates for different languages. Still I would also prefer that MM3 would assume that all template text are UTF-8, independent from the language.
- You have not customized the German template. In that case the user should get the default English template.
How do those sound to you, as goals for Mailman behavoir?
So ISTM this is a Mailman issue, if Mailman is sending out the German message with question marks substituted for non-ASCII characters, or if Mailman is sending out the German message with US-ASCII as the Content-Type charset parameter. The only way I can see this as a user (admin) issue is if you overwrote the English template with German (and even then, it's nuts for Mailman not to use UTF-8 for English what with directed quotes, emoji, and all the other enhancements to the character repertoire that US English speakers use).
Can you tell me a little more about whether you configured any custom templates, and if so, how? That is, did you use the Postorius text field or overwrite files directly?
I am installing download URLs pointing to a Nextcloud via the REST API described here:
https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/t...
However, I did not care to use URL placeholders for the language which probably I should have done. However, the RTD page only talks here about the list's preferred language, not the user's preferred language.
What strikes me more that for the localization of the auto-response subjects the global setting default_language of the server seems to be relevant and not the particular list's preferred language.
Claus
Steve
Footnotes: [1] Note to self: perhaps the admin UI can check the list of supported languages and warn if some but not all are customized.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- Claus-Justus Heine himself@claus-justus-heine.de http://www.claus-justus-heine.de/
Schatzmeister der Camerata Academica Freiburg e.V. --- www.cafev.de
On 4/25/22 02:52, Claus-Justus Heine via Mailman-users wrote:
I have installed custom templates using the REST API and pointing MM3 to some download URIs. So MM3 has custom templates for particular lists, but I did not yet install different templates for different languages. Still I would also prefer that MM3 would assume that all template text are UTF-8, independent from the language.
I think in this case the issue is not in the retrieval of the template text, but rather in the representation of the resultant message in the character set of the lists preferred language, us-ascii for 'en' in this case.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Claus-Justus Heine -
Mark Sapiro -
Stephen J. Turnbull