[MM3-users] Re: hyperkitty: "incorrect string value" for botched utf-8 encoding

27 May 2020


      On 5/25/20 9:47 AM, thoralf via Mailman-users wrote:
...
hi,
my hyperkitty installation (3.2.1 from debian buster) started to act up
recently with
(1366, "Incorrect string value: '\\xF0\\x9F\\x99\\x82\\x0AA...' for
column mailman3web.hyperkitty_email.content at row 1")
for every new mail that is about to be archived, i'll quote the full
error mail at the end of this mail.
i traced the issue to a slighty smiling face (aka u+1f642) in this
rather interestingly encoded email:
[…]
Content-Type: multipart/mixed;
boundary="===============2457401976140816263=="
--===============2457401976140816263==
Content-Type: multipart/alternative; boundary="5ecb6fd9_327b23c6_2042"
--5ecb6fd9_327b23c6_2042
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
kann halt NIE am mittwoch. viel spass =F0=9F=99=82
[…]
--5ecb6fd9_327b23c6_2042
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
<html xmlns=3D=22http://www.w3.org/1999/xhtml=22>
[…]
<div dir=3D=22auto=22>kann halt NIE am mittwoch. viel spass =F0=9F=99=82<=
/div>
--5ecb6fd9_327b23c6_2042--
[…]
--===============2457401976140816263==--
… which still lingers around in
/var/lib/mailman3/archives/hyperkitty/spool .
is this to be expected, or might it be a good idea to amend mailman3 /
hyperkitty to deal with this more gracefully?
When HyperKitty gets an exception like this in archiving a message, it
queues it in /var/lib/mailman3/archives/hyperkitty/spool and on every
new post, retries the queued messages. This message fails again on retry
and is requeued.
How would you suggest this be handled more gracefully?
The underlying issue has been seen before. The basic problem is in MySQL
and MariaDB the encoding is utf8 which only accepts utf8 encodings up to
3 bytes. It needs to be utf8mb4 to accept characters outside the base plane.
I know there is a thread on this somewhere, but I can't find it right now.
--
Mark Sapiro <mark@msapiro.net>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan