Non-ascii characters missing from Pipermail archive txt and gz downloads
Mailman 2.1.34 Debian 10 Postfix
Hi
I'm hoping someone can shine a light on character encoding issue I've encountered.
A plain-text email with non-ascii characters in the body gets posted to the list.
As per Mark Sapiro's guide I've captured the incoming message to file.
The message is received by Mailman with the non-ascii characters displaying correctly.
The header of that message has:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 8bit
In the list's mbox file and archive webpage, the message displays the non-ascii characters correctly.
In the archive's downloaded .txt (and also .gz) file, the non-ascii characters are missing and displayed as "?".
I've copied the message text in below, from both the correct one from the email and the erroneous .txt file. Hopefully they won't get scrambled up when I send this.
Any advice on getting the non-ascii characters written into the archive .txt file would be gratefully received.
Thanks, Mark
=== Message text as okay in mbox and as shown on the archive webpage ===
If one goes by the definition of veḷippaṭai as given in the Tamil Lexicon that the meaning of an ambiguous word should be disambiguated by a qualifying word, then aruvi āmpal does not conform to that definition since in the case of aruvi āmpal in Patiṟṟuppattu 63, aruvi is really made up of aru+vi, a compound. Moreover, the expression aṭai aṭuppu aṟiyā is already there to clarify that āmpal is a number and not a flower. Thus, aruvi simply provides information in addition to aṭai aṭuppu aṟiyā that āmpal is not a flower. The modern commentator Aruḷampalavaṉār also does not call it veḷippaṭai.
===
=== Message text with missing characters in te archive's txt and gz downloads ==
If one goes by the definition of ve?ippa?ai as given in the Tamil Lexicon that the meaning of an ambiguous word should be disambiguated by a qualifying word, then aruvi ?mpal does not conform to that definition since in the case of aruvi ?mpal in Pati??uppattu 63, aruvi is really made up of aru+vi, a compound. Moreover, the expression a?ai a?uppu a?iy? is already there to clarify that ?mpal is a number and not a flower. Thus, aruvi simply provides information in addition to a?ai a?uppu a?iy? that ?mpal is not a flower. The modern commentator Aru?ampalava??r also does not call it ve?ippa?ai.
===
Hi Mark
I'm not sure, but I think the problems are to do with the program you are using to view the txt file.
A modern mail reader understands the Content-type: header and will adjust its character processing accordingly, but a .txt file has no default character encoding, so any text editor will have to "use its best judgement". Some will default to utf-8, because that is compatible with 7-bit ASCII (NOT Latin1) while others will just put characters out and hope for the best (esentially leaving the result to the encoding of the font).
In theory, a UTF-encoded text file can begin with the "BOM" marker, a sequence of characters which is supposed to indicate which variety of Unicode is in use, but this is rarely present, especially for UTF-8.
Try looking at your text file with a UTF-8 capable text editor **and ensure that you tell the editor to use the UTF-8 encoding**. I expect it will look ok then.
Hope this helps,
Ruth
On 09/04/2021 13:52, Mark Dale via Mailman-users wrote:
Mailman 2.1.34 Debian 10 Postfix
Hi
I'm hoping someone can shine a light on character encoding issue I've encountered.
A plain-text email with non-ascii characters in the body gets posted to the list.
As per Mark Sapiro's guide I've captured the incoming message to file.
The message is received by Mailman with the non-ascii characters displaying correctly.
The header of that message has:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 8bit
In the list's mbox file and archive webpage, the message displays the non-ascii characters correctly.
In the archive's downloaded .txt (and also .gz) file, the non-ascii characters are missing and displayed as "?".
I've copied the message text in below, from both the correct one from the email and the erroneous .txt file. Hopefully they won't get scrambled up when I send this.
Any advice on getting the non-ascii characters written into the archive .txt file would be gratefully received.
Thanks, Mark
=== Message text as okay in mbox and as shown on the archive webpage ===
If one goes by the definition of veḷippaṭai as given in the Tamil Lexicon that the meaning of an ambiguous word should be disambiguated by a qualifying word, then aruvi āmpal does not conform to that definition since in the case of aruvi āmpal in Patiṟṟuppattu 63, aruvi is really made up of aru+vi, a compound. Moreover, the expression aṭai aṭuppu aṟiyā is already there to clarify that āmpal is a number and not a flower. Thus, aruvi simply provides information in addition to aṭai aṭuppu aṟiyā that āmpal is not a flower. The modern commentator Aruḷampalavaṉār also does not call it veḷippaṭai.
===
=== Message text with missing characters in te archive's txt and gz downloads ==
If one goes by the definition of ve?ippa?ai as given in the Tamil Lexicon that the meaning of an ambiguous word should be disambiguated by a qualifying word, then aruvi ?mpal does not conform to that definition since in the case of aruvi ?mpal in Pati??uppattu 63, aruvi is really made up of aru+vi, a compound. Moreover, the expression a?ai a?uppu a?iy? is already there to clarify that ?mpal is a number and not a flower. Thus, aruvi simply provides information in addition to a?ai a?uppu a?iy? that ?mpal is not a flower. The modern commentator Aru?ampalava??r also does not call it ve?ippa?ai.
===
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- Software Manager & Engineer Tel: 01223 414180 Blog: http://www.ivimey.org/blog LinkedIn: http://uk.linkedin.com/in/ruthivimeycook/
Hi Ruth,
Apologies for the late response. I posted my message to the MM3 list in error and have since reposted it to the MM2 list.
However, thank you for taking the trouble to reply.
The problem lies outside the choice of editor as both files are being viewed in the same editor -- with one correct and the other not.
Mark Sapiro has explained (on the MM2 list) what is causing the problem. Armed with that information I just have to now find a way ....
Regards, Mark
-------- Original Message -------- From: Ruth Ivimey-Cook [mailto:ruth@ivimey.org] Sent: Friday, April 9, 2021, 15:24 UTC To: mark@postmark.io, Mailman users Subject: [MM3-users] Non-ascii characters missing from Pipermail archive txt and gz downloads
Hi Mark
I'm not sure, but I think the problems are to do with the program you are using to view the txt file.
A modern mail reader understands the Content-type: header and will adjust its character processing accordingly, but a .txt file has no default character encoding, so any text editor will have to "use its best judgement". Some will default to utf-8, because that is compatible with 7-bit ASCII (NOT Latin1) while others will just put characters out and hope for the best (esentially leaving the result to the encoding of the font).
In theory, a UTF-encoded text file can begin with the "BOM" marker, a sequence of characters which is supposed to indicate which variety of Unicode is in use, but this is rarely present, especially for UTF-8.
Try looking at your text file with a UTF-8 capable text editor **and ensure that you tell the editor to use the UTF-8 encoding**. I expect it will look ok then.
Hope this helps,
Ruth
On 09/04/2021 13:52, Mark Dale via Mailman-users wrote:
Mailman 2.1.34 Debian 10 Postfix
Hi
I'm hoping someone can shine a light on character encoding issue I've encountered.
A plain-text email with non-ascii characters in the body gets posted to the list.
As per Mark Sapiro's guide I've captured the incoming message to file.
- https://www.mail-archive.com/mailman-users@python.org/msg73469.html
The message is received by Mailman with the non-ascii characters displaying correctly.
The header of that message has:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 8bit
In the list's mbox file and archive webpage, the message displays the non-ascii characters correctly.
In the archive's downloaded .txt (and also .gz) file, the non-ascii characters are missing and displayed as "?".
I've copied the message text in below, from both the correct one from the email and the erroneous .txt file. Hopefully they won't get scrambled up when I send this.
Any advice on getting the non-ascii characters written into the archive .txt file would be gratefully received.
Thanks, Mark
=== Message text as okay in mbox and as shown on the archive webpage ===
If one goes by the definition of veḷippaṭai as given in the Tamil Lexicon that the meaning of an ambiguous word should be disambiguated by a qualifying word, then aruvi āmpal does not conform to that definition since in the case of aruvi āmpal in Patiṟṟuppattu 63, aruvi is really made up of aru+vi, a compound. Moreover, the expression aṭai aṭuppu aṟiyā is already there to clarify that āmpal is a number and not a flower. Thus, aruvi simply provides information in addition to aṭai aṭuppu aṟiyā that āmpal is not a flower. The modern commentator Aruḷampalavaṉār also does not call it veḷippaṭai.
===
=== Message text with missing characters in te archive's txt and gz downloads ==
If one goes by the definition of ve?ippa?ai as given in the Tamil Lexicon that the meaning of an ambiguous word should be disambiguated by a qualifying word, then aruvi ?mpal does not conform to that definition since in the case of aruvi ?mpal in Pati??uppattu 63, aruvi is really made up of aru+vi, a compound. Moreover, the expression a?ai a?uppu a?iy? is already there to clarify that ?mpal is a number and not a flower. Thus, aruvi simply provides information in addition to a?ai a?uppu a?iy? that ?mpal is not a flower. The modern commentator Aru?ampalava??r also does not call it ve?ippa?ai.
===
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
participants (2)
-
Mark Dale -
Ruth Ivimey-Cook