Hyperkitty Text/Markdown/HTML
How about a feature to include "HTML" in the current Hyperkitty choices of (Text or Markdown) rendering?
A sanitizer must be used to prevent malicious attacks. https://github.com/messense/nh3 . lxml is another package that processes html.
A sanitizer would be helpful for more aspects. Permit a limited set of markdown so that messages still appear professional. Gitlab/Github don't allow arbitrary fonts or font sizes, right? Mostly remove these tags.
A way that code blocks could be presented: determine an established list of known codeblock styles. Configure the sanitizer to permit this range of fonts or styling elements. By allowing them through the filter, a code styling is rendered. (It is permitted/allowed to be rendered.) Even better, when the sanitizer recognizes the fonts, it replaces them with an "official" set of codeblock fonts.
Otherwise, if other font tags are entirely removed, if they are missing, the email client or web ui will decide the font selection, and it will appear "standard".
In order for both email clients and Hyperkitty to view the same result, the sanitizer should ideally be applied to outgoing emails as well as rendering in the UI. That means this feature would affect more than just Hyperkitty. (i.e. mailman-core). Although, that could be implemented as a separate step, until which time emails would get a full range of HTML, without sanitizing.
In this scheme, ensure outgoing emails are multi-part MIME with plain text and html versions.
Some of our users are really in favor of more formatting.
Others prefer plain text.
Those who prefer plain text have various arguments that HTML messages will be rendered badly, and won't be legible.
Questions:
- Would "multi-part MIME with plain text and html versions both" solve the problem for users who prefer plain text?
- What percentage of email clients that support HTML formatting when sending will also include a plain text rendering of the same content in the outgoing message?
- Is it usually the responsibility of a receiving mail client to be able to convert HTML to plain text if the user requests it?
- When mailman-core receives and then sends out a message, can it receive HTML, generate a plain-text version, and then send both?
Thoughts?
On 9/8/25 07:22, Sam Darwin via Mailman-users wrote:
How about a feature to include "HTML" in the current Hyperkitty choices of (Text or Markdown) rendering?
I don't understand what you envision such a rendering to do. First of all, this setting only affects rendering in HyperKitty thread and message views of archived messages. It has nothing at all to do with messages delivered to list members.
Archived messages which contain text/html parts currently render those parts as attachments that can be downloaded. Are you suggesting that "HTML" rendering would render those inline or something else?
A sanitizer must be used to prevent malicious attacks. https://github.com/messense/nh3 . lxml is another package that processes html.
A sanitizer would be helpful for more aspects. Permit a limited set of markdown so that messages still appear professional. Gitlab/Github don't allow arbitrary fonts or font sizes, right? Mostly remove these tags.
A way that code blocks could be presented: determine an established list of known codeblock styles. Configure the sanitizer to permit this range of fonts or styling elements. By allowing them through the filter, a code styling is rendered. (It is permitted/allowed to be rendered.) Even better, when the sanitizer recognizes the fonts, it replaces them with an "official" set of codeblock fonts.
Otherwise, if other font tags are entirely removed, if they are missing, the email client or web ui will decide the font selection, and it will appear "standard".
Your reference to email client suggests you are thinking in terms of messages delivered to list members. HyperKitty's rendering mode has nothing to do with that.
In order for both email clients and Hyperkitty to view the same result, the sanitizer should ideally be applied to outgoing emails as well as rendering in the UI. That means this feature would affect more than just Hyperkitty. (i.e. mailman-core). Although, that could be implemented as a separate step, until which time emails would get a full range of HTML, without sanitizing.
As far as Mailman core is concerned, Things like this are handled in content filtering. In particular the settings Collapse alternatives and Convert html to plaintext.
In this scheme, ensure outgoing emails are multi-part MIME with plain text and html versions.
If the incoming message is multipart/alternative with text/plain and text/html alternatives and content filtering accepts all those MIME types and does not Collapse alternatives, the outgoing message will be the same.
If you're suggesting a text/html part outside of a multipart/alternative part should cause Mailman to create a text/plain alternative and convert the text/html part into a multipart/alternative part containing the text/plain and text/html alternatives, that could be done, but I'm not interested in doing it. We already have a mechanism for converting HTML to plain text, but current code replaces the HTML.
As far as converting plain text to HTML, the straight forward way to to that is just to wrap it in <pre>..</pre> tags, but there seems to be nothing gained by that. As far as converting urls and email addresses into clickable links, many MUAs already do that.
Some of our users are really in favor of more formatting.
Others prefer plain text.
Those who prefer plain text have various arguments that HTML messages will be rendered badly, and won't be legible.
All of the settings mentioned are per list, not user option/
Questions:
- Would "multi-part MIME with plain text and html versions both" solve the problem for users who prefer plain text?
If they set their MUA to render the text/plain alternative.
- What percentage of email clients that support HTML formatting when sending will also include a plain text rendering of the same content in the outgoing message?
Many if not all mainstream ones do or at least have settings to control it, but some do a horrible job. For example if in Yahoo mail one composes in "rich text" and drags and drops or copy/pastes or in some ways even types a link, the generated text/plain alternative doesn't contain the url in any form - see <https://www.grizz.org/wiki/PostingURLsToLists>
- Is it usually the responsibility of a receiving mail client to be able to convert HTML to plain text if the user requests it?
If a message part is multipart/alternative and the user wants plain text, I think most MUAs will just display the text/plain alternative. If the message is HTML only, I suppose the MUA will convert it somehow.
- When mailman-core receives and then sends out a message, can it receive HTML, generate a plain-text version, and then send both?
Not currently. It can convert the HTML to plain text using a configured process such as lynx or elinks, but then only the plain text is sent, not both.
On the other hand, if the incoming message is multipart/alternative and content filtering is off or appropriately set, the outgoing message will be multipart/alternative.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I don't understand what you envision such a rendering to do.
To achieve a level of rendering emails (and web) similar to gitlab or github. That includes bold, italics, code blocks, and possibly more. In both Hyperkitty and email clients.
It has nothing at all to do with messages delivered to list members.
that's why I mentioned "would affect more than just Hyperkitty. (i.e. mailman-core)."
Archived messages which contain text/html parts currently render those parts as attachments that can be downloaded. Are you suggesting that "HTML" rendering would render those inline or something else?
Yes "inline", as part of the main message display, and not as an attachment. No attachment would appear.
HyperKitty's rendering mode has nothing to do with that.
Hyperkitty is one of the two main components. It's the web UI part. The other being the outgoing emails.
If you're suggesting a text/html part outside of a multipart/alternative part
I was probably imagining just "multipart/alternative", containing alternatives. Is the "outside" method also a possibility? I did not intend that.
For example if in Yahoo mail one composes in "rich text" and drags and drops or copy/pastes or in some ways even types a link, the generated text/plain alternative doesn't contain the url in any form.
Our lists are set to "Convert html to plaintext". I just tested from Yahoo email, with rich text, and hyperlinks. Everything was fine. The problem didn't exhibit itself. I also "copied" a link from a website, and pasted it into the email, but it was ok. Perhaps other clients "aol.com, att.net, netscape.net, pacbell.net, prodigy.net, rocketmail.com, sbcglobal.net and ymail.com" have a problem? But today in mail.yahoo.com , no...
If most mail clients include Yahoo and Gmail send both HTML and plain-text versions in a multipart message, there would theoretically be two choices for mm3 with "Convert html to plaintext".
- Mail out the included plain-text copy from the multipart email. Or,
- Parse the HTML part of the message and "convert html to plaintext".
which of those is occurring?
To summarize again, the goal is to show in Hyperkitty and in email clients a format (based on HTML) that includes bold, italics, bullet items, and code blocks. The same way github or gitlab messages appear, with more formatting options. Since HTML is a standard format, and many mail clients have built-in support for both generating and displaying it, HTML is nature choice for a protocol to use, it's already included everywhere.
On 9/8/25 13:20, Sam Darwin via Mailman-users wrote:
I don't understand what you envision such a rendering to do.
To achieve a level of rendering emails (and web) similar to gitlab or github. That includes bold, italics, code blocks, and possibly more. In both Hyperkitty and email clients.
If an incoming message contains HTML and it is not filtered or replaced by content filtering, it will be in the outgoing message and the message sent to HyperKitty. Thus, all you are asking here is that there be an option for HyperKitty to render the HTML (perhaps sanitized) inline as opposed to leaving it as an attachment. I understand that request.
Hyperkitty is one of the two main components. It's the web UI part. The other being the outgoing emails.
Actually Mailman core is the main component. Postorius and HyperKitty are really just examples of web based list management and archive rendering applications that communicate with Mailman core via it's REST API. Others have been developed, e.g. EMWD's Affinity and Empathy applications. This is really just and aside, but my point is that Postorius and HyperKitty are not integral parts of Mailman.
If you're suggesting a text/html part outside of a multipart/alternative part
I was probably imagining just "multipart/alternative", containing alternatives. Is the "outside" method also a possibility? I did not intend that.
I was asking if you envisioned that a text/html part outside of a multipart/alternative part would cause mailman to create a text/plain alternative and package it with the text/html part in a multipart/alternative part. But if that was not your intent, OK.
For example if in Yahoo mail one composes in "rich text" and drags and drops or copy/pastes or in some ways even types a link, the generated text/plain alternative doesn't contain the url in any form.
Our lists are set to "Convert html to plaintext". I just tested from Yahoo email, with rich text, and hyperlinks. Everything was fine. The problem didn't exhibit itself. I also "copied" a link from a website, and pasted it into the email, but it was ok. Perhaps other clients "aol.com, att.net, netscape.net, pacbell.net, prodigy.net, rocketmail.com, sbcglobal.net and ymail.com" have a problem? But today in mail.yahoo.com , no...
The issue is not in the plain text converted from the HTML by Mailman's Convert html to plaintext feature. The issue is in the text/plain alternative created by Yahoo when a message is composed as "rich text" and created by Yahoo as multipart/alternative. Were you looking at that text/plain alternative part from Yahoo., i.e. what you see in the outgoing mail if Collapse alternatives is Yes.
If most mail clients include Yahoo and Gmail send both HTML and plain-text versions in a multipart message, there would theoretically be two choices for mm3 with "Convert html to plaintext".
- Mail out the included plain-text copy from the multipart email. Or,
- Parse the HTML part of the message and "convert html to plaintext". which of those is occurring?
1 occurs if content filtering Collapse alternatives is Yes.
2 occurs if Convert html to plain text is yes, but the result depends on Collapse alternatives. If Collapse alternatives is Yes only the original text/plain alternative will be in the outgoing mail. If Collapse alternatives is No, both the original text/plain alternative and a second text/plain alternative consisting of the Mailman converted HTML will be in the outgoing messages multipart/alternative part, and most MUAs will render the converted html alternative.
To summarize again, the goal is to show in Hyperkitty and in email clients a format (based on HTML) that includes bold, italics, bullet items, and code blocks.
If you want email clients to display the HTML, either don't filter content or filter content and pass both multipart and text main types and set both Collapse alternatives and Convert html to plaintext to No.
This will accomplish what you want minus the possible sanitizing of the HTML for Mailman core and delivered messages. In HyperKitty, the HTML part(s) will still be rendered as attachments.
If we receive a merge request to implement inline rendering of HTML, it will be considered but no guarantees.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thus, all you are asking here is that there be an option for HyperKitty to render the HTML (perhaps sanitized) inline as opposed to leaving it as an attachment. I understand that request.
Yes.
Or, the more the topic is discussed, I see there are two parts. Hyperkitty rendering is one of them. The other being to find a way to sanitize HTML of outgoing emails.
I was asking if you envisioned that a text/html part outside of a multipart/alternative part would cause mailman to create a text/plain alternative and package it with the text/html part in a multipart/alternative part. But if that was not your intent, OK.
I hadn't thought about that situation.
Maybe it is relevant after all. And then, you said that you are not interested in doing that. It might need to be investigated along with these other features. Not sure.
The issue is not in the plain text converted from the HTML by Mailman's Convert html to plaintext feature. The issue is in the text/plain alternative created by Yahoo when a message is composed as "rich text" and created by Yahoo as multipart/alternative. Were you looking at that text/plain alternative part from Yahoo., i.e. what you see in the outgoing mail if Collapse alternatives is Yes.
The list's settings are Collapse alternatives=Yes and Convert html to plaintext=YES. Standard "plain" URLs are ok, but I was just now able to replicate some sort of problem. Using the yahoo toolbar, a fully HTML hyperlink with an href and name failed to render in plain text. It resulted in just plain text without appearing as a "link".
If that's happening even now, maybe it won't get worse in the future. only better. if switching to html.
and most MUAs will render the converted html alternative.
interesting... The future plan would be to set all filtering to NO. All MIME parts delivered. But then, what if a plain-text version is missing somehow.
If you want email clients to display the HTML, either don't filter content or filter content and pass both multipart and text main types and set both Collapse alternatives and Convert html to plaintext to No. This will accomplish what you want minus the possible sanitizing of the HTML for Mailman core and delivered messages. In HyperKitty, the HTML part(s) will still be rendered as attachments.
Understood. Problems remain though: "minus sanitizing". Sanitizing should be added. "HTML part(s) will still be rendered as attachments." The bulk of this discussion is about avoiding Hyperkitty attachments, and instead showing html inline.
If we receive a merge request to implement inline rendering of HTML, it will be considered but no guarantees.
Exploring the issue.
On 9/8/25 15:38, Sam Darwin via Mailman-users wrote:
Or, the more the topic is discussed, I see there are two parts. Hyperkitty rendering is one of them. The other being to find a way to sanitize HTML of outgoing emails.
If you want to accept HTML in posts, you should ensure that only trusted users can post. This is good practice anyway to avoid spam on the lists. Then you don't have to be too concerned about malicious HTML.
I was asking if you envisioned that a text/html part outside of a multipart/alternative part would cause mailman to create a text/plain alternative and package it with the text/html part in a multipart/alternative part. But if that was not your intent, OK.
I hadn't thought about that situation.
Maybe it is relevant after all. And then, you said that you are not interested in doing that. It might need to be investigated along with these other features. Not sure.
You can currently convert HTML to plain text, but that replaces the HTML. I understand you want both so your users have the option of seeing one or the other depending on MUA settings, but I don't see a demand for a feature like that.
The issue is not in the plain text converted from the HTML by Mailman's Convert html to plaintext feature. The issue is in the text/plain alternative created by Yahoo when a message is composed as "rich text" and created by Yahoo as multipart/alternative. Were you looking at that text/plain alternative part from Yahoo., i.e. what you see in the outgoing mail if Collapse alternatives is Yes.
The list's settings are Collapse alternatives=Yes and Convert html to plaintext=YES. Standard "plain" URLs are ok, but I was just now able to replicate some sort of problem. Using the yahoo toolbar, a fully HTML hyperlink with an href and name failed to render in plain text. It resulted in just plain text without appearing as a "link".
If you are saying it resulted in the URL being rendered in the text/plain alternative as a text string, that is not what I see.
I just composed a message from Yahoo to a non-yahoo address. I created the message in Yahoo's HTML editor. It simply said "Hi mark. Here's a link." after which I pasted today's Google Doodle link. In the resultant message the HTML part rendered as
Hi mark. Here's a link. Google Search
where "Google Search" was the above link I had pasted and this was also followed by a Google Search graphic which was also the same link.
However the text/plain part in its entirety (minus my signature) was
Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi mark. Here's a link. Google Search
|=20 |=20 | |=20 Google Search
|
|
|
The display text "Google Search" was in the text plain part, but the text of the link itself appeared nowhere in the text/plain part
interesting... The future plan would be to set all filtering to NO. All MIME parts delivered. But then, what if a plain-text version is missing somehow.
Then the message will contain only the HTML. But if you do that, malicious HTML and missing plain text will be the least of the things to be concerned about. All sorts of malicious malware can be attached to email in other than HTML parts.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
If you want to accept HTML in posts, you should ensure that only trusted users can post. This is good practice anyway to avoid spam on the lists. Then you don't have to be too concerned about malicious HTML.
Right, that is already the case. However equally important is formatting via the sanitizer, not just removing malicious HTML.
I don't see a demand for a feature like that.
Currently there is no demand for that, but also currently Hyperkitty only supports plaintext or markdown, so if you add HTML into the mix, it may change requirements.
Yahoo... that is not what I see.
We both see the actual link is missing. The difference is, that you also are getting some garbage characters (multiple "|=20"), and I did not see that, although maybe I was not pasting the right thing.
Then the message will contain only the HTML. But if you do that, malicious HTML and missing plain text will be the least of the things to be concerned about. All sorts of malicious malware can be attached to email in other than HTML parts.
You are saying "All sorts of malicious malware can be attached to email in other than HTML parts." What is the solution to this currently? We have "Collapse alternatives" and "Convert html to plaintext" enabled. If one of those is relevant (is it?) the description in postorius should not be limited to "Should Mailman collapse multipart?" but also say "This is strongly recommended, to remove all sorts of malicious malware", since that seems critical. Or, which postorius setting applies? Thanks.
Sam Darwin via Mailman-users writes:
Right, that is already the case. However equally important is formatting via the sanitizer, not just removing malicious HTML.
Not clear what "sanitization" you want to suggest here. Authors want HTML so they can have more control over the presentation than offered by Markdown (which most agents don't implement anyway). Sanitizing HTML likely breaks the intended presentation. Fundamental features of HTML such as links and automatic display of multimedia objects are obvious vulnerabilities, but disabling those will not make your HTML-loving users happy!
You are saying "All sorts of malicious malware can be attached to email in other than HTML parts." What is the solution to this currently?
Discard them or save them as attachments, usually as .bin with a non-specific MIME type such as application/octet-stream. Also, these are image/* and application/* parts, and malware filters run by the MTA know quite well what to do with them. The problem with HTML is that it's more amenable to social engineering than most such formats (PDF is an obvious exception).
We have "Collapse alternatives" and "Convert html to plaintext" enabled. If one of those is relevant
Relevant to possibly malicious email? As far as the documentation goes, maybe it should be changed to mention that.
But the default for "Collapse alternatives" is True. In the past most HTML mail was bundled with a plain text alternative, so that would take care of possibly malicious HTML parts. OTOH, assuming that HTML- only mail was deliberately configured that way by the sender, we did set the default for "Convert HTML to plaintext" to False to respect that intent, as it's likely that there is legitimate content that can't be represented as plaintext or attractively presented as a separate MIME part. I think that logic is still probably valid. And as Mark said, the likelihood that malicious mail will be sent via a mailing list is relatively low as long as it's set to members-only posting. So the current default settings seem like a good compromise, and it's not obvious to me that it's worth discouraging changing those settings if they're in demand from the subscribers and the list owner is OK with that.
For Mailman the real problem is that doing more than we currently do, and doing it well, would require construction of the DOM and a pretty capable CSS implementation, with features driven by what the full range of user agents implement. It's a lot of work, and it's not obvious what the benefit to most Mailman users would be, over and above offering "collapse = False; convert = False" as an option and defaulting to "collapse = True".
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan
the benefit to most Mailman users would be
Imagine the user experience when reading/writing a github issue and/or getting an email about it. There is some formatting, even if limited, such as code blocks. Users appreciate that in technical code discussions, and so it's a benefit. They may think "Why can't I use italics?" Allow basic formatting. Remove colors, font sizing, for example.
construction of the DOM and a pretty capable CSS implementation
Basic html tags here are bold, italics, hyperlinks, code blocks... if these were allowed through a new html sanitizer layer, and could then be presented to the reader on Hyperkitty, I don't see why that would require anything advanced in terms of "construction of the DOM and a pretty capable CSS implementation". Just send those html tags to the browser, as-is. Perhaps nothing needs to change in terms of CSS or DOM.
On 9/9/25 13:39, Sam Darwin via Mailman-users wrote:
Imagine the user experience when reading/writing a github issue and/or getting an email about it. There is some formatting, even if limited, such as code blocks. Users appreciate that in technical code discussions, and so it's a benefit. They may think "Why can't I use italics?" Allow basic formatting. Remove colors, font sizing, for example.
Both GitHub and GitLab use Markdown formatting. So does HyperKitty if you enable it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Both GitHub and GitLab use Markdown formatting. So does HyperKitty if you enable it.
Problems with the Markdown implementation, when testing:
- Renders in Hyperkitty, but not emails.
- bold and italics are displayed, but still include the asterisks also?
In favor of HTML, most sending and receiving email clients already offer HTML support, if you are composing an email in Gmail, and then receiving it in Yahoo, there are already HTML formatting options.
participants (3)
-
Mark Sapiro -
Sam Darwin -
Stephen J. Turnbull