I use Lithuanian language (it has special characters like ąčįū...) for regular message template feature. I noticed that some posts come correct, but some have question marks (?) in places of these characters. Looks like there is sittings trick. But footers adds mailman, so it should be always correct, or it depends on senders environment setup?
Kind regards Ugnius
On 9/12/19 1:16 PM, Ugnius wrote:
I use Lithuanian language (it has special characters like ąčįū...) for regular message template feature. I noticed that some posts come correct, but some have question marks (?) in places of these characters. Looks like there is sittings trick. But footers adds mailman, so it should be always correct, or it depends on senders environment setup?
Have you set your list's preferred language to Lithuanian (lt)? If so, Mailman's character set for that language is iso-8859-13. If the list's preferred language is English, Mailman's character set for that language is us-ascii. These are artifacts from Mailman 2.1 and should ultimately be changed to utf-8, but in the mean time, templates need to be encoded in the appropriate character set.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Have you set your list's preferred language to Lithuanian (lt)? If so, Mailman's character set for that language is iso-8859-13. Yes, I have made several tests. I have set charset: iso-8859-13 and default_language: lt in mailman.cfg. (Also I have set LANGUAGE_CODE = 'lt' in the settings_local.py, but I think it is not related to this case.) I have tried to set utf-8 as well in mailman.cfg as well. And I can confirm - in all tested cases text characters are correct (in footer as well) if sender sends text only. If sender sends message in HTML (mostly all users send) then no matter what the encoding is set in my mailman's configuration or iso-8859-13, or utf-8 message body looks correct, but in
Hi, Mark. footer all national characters are replaced with question marks " ? ". I would not like to use HTML to TEXT converting, like users like to combine text and pictures in messages. There are some more LT charsets. Windows: "Windows-1257" and Oracle use "blt8mswin1257" and there is older standard: "iso-8859-4". Actually currently problem are messages in HTML.
On 9/14/19 4:06 AM, Ugnius S wrote:
Yes, I have made several tests. I have set charset: iso-8859-13 and default_language: lt in mailman.cfg.
If you created the list before you set default_language: lt in mailman.cfg, the list's preferred_language is possibly 'en;' with character set us-ascii.
(Also I have set LANGUAGE_CODE = 'lt' in the settings_local.py, but I think it is not related to this case.)
Yes, LANGUAGE_CODE in settings_local.py is not relevant to this.
I have tried to set utf-8 as well in mailman.cfg as well. And I can confirm - in all tested cases text characters are correct (in footer as well) if sender sends text only.
OK
If sender sends message in HTML (mostly all users send) then nomatter what the encoding is set in my mailman's configuration or iso-8859-13, or utf-8 message body looks correct, but in footer all national characters are replaced with question marks " ? ".
The addition of the footer is done by mailman/handlers/decorate.py. In the plain text case the message body is converted to unicode, the unicode footer is appended to the body and the new body is encoded for transmission. The encoding is the first of the list's character set, the incoming message's character set or utf-8 which produces no error.
In the case where the incoming message is not a single plain text message part, the footer is added as a separate MIME part with the unicode footer encoded in the list's character set with errors='replace'. This is what produces the '?' characters. I suspect this happens because the list's preferred_language is 'en' with character set us-ascii.
try running Mailman's
bin/mailman shell -l your_list@domain
at the prompt enter
m.preferred_language
if the response is not
<Language [lt] Lithuanian>
do
m.preferred_language = 'lt'
and then control-D to exit.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark, You were 100% right for fixing. I was not sure what the language setting were on list creating, but one of the lists was created a day ago and I was doubt about it a little. So I made test: I have checked that default language is lt, and then I created the new list for testing. I had language issues with the HTML as well. Your recipe helped to resolve the issue. :) I fixed all my lists. I have stopped the mailman (just in case), than executed as You had told:
$ mailman shell -l test@example.com Welcome to the GNU Mailman shell The variable 'm' is the test@example.com mailing list
m.preferred_language <Language [en] English (USA)> m.preferred_language = 'lt'
<Ctrl>+D
Thank You very much. This fix made magic.
Kind regards Ugnius
2019-09-14, št, 20:20 Mark Sapiro <mark@msapiro.net> rašė:
On 9/14/19 4:06 AM, Ugnius S wrote:
Yes, I have made several tests. I have set charset: iso-8859-13 and default_language: lt in mailman.cfg.
If you created the list before you set default_language: lt in mailman.cfg, the list's preferred_language is possibly 'en;' with character set us-ascii.
(Also I have set LANGUAGE_CODE = 'lt' in the settings_local.py, but I think it is not related to this case.)
Yes, LANGUAGE_CODE in settings_local.py is not relevant to this.
I have tried to set utf-8 as well in mailman.cfg as well. And I can confirm - in all tested cases text characters are correct (in footer as well) if sender sends text only.
OK
If sender sends message in HTML (mostly all users send) then nomatter what the encoding is set in my mailman's configuration or iso-8859-13, or utf-8 message body looks correct, but
in
footer all national characters are replaced with question marks " ? ".
The addition of the footer is done by mailman/handlers/decorate.py. In the plain text case the message body is converted to unicode, the unicode footer is appended to the body and the new body is encoded for transmission. The encoding is the first of the list's character set, the incoming message's character set or utf-8 which produces no error.
In the case where the incoming message is not a single plain text message part, the footer is added as a separate MIME part with the unicode footer encoded in the list's character set with errors='replace'. This is what produces the '?' characters. I suspect this happens because the list's preferred_language is 'en' with character set us-ascii.
try running Mailman's
bin/mailman shell -l your_list@domain
at the prompt enter
m.preferred_language
if the response is not
<Language [lt] Lithuanian>
do
m.preferred_language = 'lt'
and then control-D to exit.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Hi. I am not sure, maybe it could be another topic about header and footer, but I have met one additional issue related to that.
This noticed only on HTML text messages. If message comes in HTML the header does not add extra <enter> new line, new paragraph and footer as well. For example if message body has no additional, list's decorated message looks next:
TEST HEADER _____________________Hello,
text text text, pictures, text
regards Ugnius____________________ TEST FOOTER
I have tried to add more empty lines in "The footer for a regular (non-digest) message.", but mailman removes these empty lines. Work around is put dot "." and then several empty lines.
Mark, I have found Your post about HTML headers and footers: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/K... YES, that works good, I have tested it, but only in cases if message comes in HTML. If sender sends plain text message, mailman adds footer with all HTML code:
<BR>__________________<BR> <B> LIST NAME </B>
Could it be possible that decorate.py would recognise message and if it has HTML text body, then just would add header/footer and if not (if message body is plain text) , then would strip header/footer "html to plain text" may removing html code from it?
Kind regards Ugnius
2019-09-14, št, 23:07 Ugnius S <ugniusviln@gmail.com> rašė:
Mark, You were 100% right for fixing. I was not sure what the language setting were on list creating, but one of the lists was created a day ago and I was doubt about it a little. So I made test: I have checked that default language is lt, and then I created the new list for testing. I had language issues with the HTML as well. Your recipe helped to resolve the issue. :) I fixed all my lists. I have stopped the mailman (just in case), than executed as You had told:
$ mailman shell -l test@example.com Welcome to the GNU Mailman shell The variable 'm' is the test@example.com mailing list
m.preferred_language <Language [en] English (USA)> m.preferred_language = 'lt'
<Ctrl>+D
Thank You very much. This fix made magic.
Kind regards Ugnius
2019-09-14, št, 20:20 Mark Sapiro <mark@msapiro.net> rašė:
On 9/14/19 4:06 AM, Ugnius S wrote:
Yes, I have made several tests. I have set charset: iso-8859-13 and default_language: lt in mailman.cfg.
If you created the list before you set default_language: lt in mailman.cfg, the list's preferred_language is possibly 'en;' with character set us-ascii.
(Also I have set LANGUAGE_CODE = 'lt' in the settings_local.py, but I think it is not related to this case.)
Yes, LANGUAGE_CODE in settings_local.py is not relevant to this.
I have tried to set utf-8 as well in mailman.cfg as well. And I can confirm - in all tested cases text characters are correct (in footer as well) if sender sends text only.
OK
If sender sends message in HTML (mostly all users send) then nomatter what the encoding is set in my mailman's configuration or iso-8859-13, or utf-8 message body looks correct, but
in
footer all national characters are replaced with question marks " ? ".
The addition of the footer is done by mailman/handlers/decorate.py. In the plain text case the message body is converted to unicode, the unicode footer is appended to the body and the new body is encoded for transmission. The encoding is the first of the list's character set, the incoming message's character set or utf-8 which produces no error.
In the case where the incoming message is not a single plain text message part, the footer is added as a separate MIME part with the unicode footer encoded in the list's character set with errors='replace'. This is what produces the '?' characters. I suspect this happens because the list's preferred_language is 'en' with character set us-ascii.
try running Mailman's
bin/mailman shell -l your_list@domain
at the prompt enter
m.preferred_language
if the response is not
<Language [lt] Lithuanian>
do
m.preferred_language = 'lt'
and then control-D to exit.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
On 9/15/19 3:33 AM, Ugnius S wrote:
This noticed only on HTML text messages. If message comes in HTML the header does not add extra <enter> new line, new paragraph and footer as well. For example if message body has no additional, list's decorated message looks next:
TEST HEADER _____________________Hello,
text text text, pictures, text
regards Ugnius____________________ TEST FOOTER
I have tried to add more empty lines in "The footer for a regular (non-digest) message.", but mailman removes these empty lines. Work around is put dot "." and then several empty lines.
The stripping of leading whitespace from templates created in Postorius is <https://gitlab.com/mailman/postorius/issues/301> to be fixed by <https://gitlab.com/mailman/postorius/merge_requests/434>
Mark, I have found Your post about HTML headers and footers: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/K... YES, that works good, I have tested it, but only in cases if message comes in HTML. If sender sends plain text message, mailman adds footer with all HTML code:
<BR>__________________<BR> <B> LIST NAME </B>
Could it be possible that decorate.py would recognise message and if it has HTML text body, then just would add header/footer and if not (if message body is plain text) , then would strip header/footer "html to plain text" may removing html code from it?
As it says in the post you reference:
message and digest footers and headers are always added either to a MIME text/plain message body or as separate MIME text/plain parts. HTML tags will just be displayed as plain text and will not be rendered as HTML.
Also see <https://wiki.list.org/x/4030707>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Mark Sapiro -
Ugnius -
Ugnius S