On 2020-02-03 20:24, Mark Sapiro wrote:

On 2/3/20 4:27 AM, Lars Schimmer wrote:

...

In a mailinglist it seems some mails are not forwarded to a member, but is listed in archive in hyperkitty. That mail did include a PDF attachement, but HyperKitty does not list any attachement.

There are two issues here. Why wasn't the message delivered to list members?, and why is there no attachment in Hyperkitty?

For the first issue, you need to examine mailman.log for the time of the post looking for errors and tracebacks. Also, the MTA logs may have useful info if the messages were sent but not delivered.

Thank you so far, I maye found the issue: Uncaught runner exception: 'ascii' codec can't encode character '\xe9' in position 109: ordinal not in range(128) Uncaught runner exception: 'ascii' codec can't encode character '\xa0' in position 37: ordinal not in range(128)

Seems to be the problemm, as it happens the same moment as the problematic email arrives in exim4:

2020-01-31 03:02:41 1ixLdp-0003Mo-5z => user@abc.de R=mailman3_router T=mailman3_transport H=localhost [127.0.0.1] C="250 Ok" 2020-01-31 03:02:41 1ixLdp-0003Mo-5z Completed

and nothing more. No sign of mailman3 sending out any mail via this list. Other emails getting to that list will be send out via user-bounces@abc.de.

Those mails are in the mailinglist archives, not on hold or anything. Just the attachement missing.

So, please, howto fix the ascii encoding error in mailman3?

Full trace: Jan 31 03:02:43 2020 (32591) Uncaught runner exception: 'ascii' codec can't encode character '\xe9' in position 109: ordinal not in range(128) Jan 31 03:02:43 2020 (32591) Traceback (most recent call last): File "/usr/lib/python3/dist-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/usr/lib/python3/dist-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/usr/lib/python3/dist-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/usr/lib/python3/dist-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/usr/lib/python3/dist-packages/mailman/handlers/to_digest.py", line 48, in process mbox.add(msg) File "/usr/lib/python3.7/mailbox.py", line 603, in add self._toc[self._next_key] = self._append_message(message) File "/usr/lib/python3.7/mailbox.py", line 758, in _append_message offsets = self._install_message(message) File "/usr/lib/python3.7/mailbox.py", line 830, in _install_message self._dump_message(message, self._file, self._mangle_from_) File "/usr/lib/python3.7/mailbox.py", line 215, in _dump_message gen.flatten(message) File "/usr/lib/python3.7/email/generator.py", line 116, in flatten self._write(msg) File "/usr/lib/python3.7/email/generator.py", line 181, in _write self._dispatch(msg) File "/usr/lib/python3.7/email/generator.py", line 214, in _dispatch meth(msg) File "/usr/lib/python3.7/email/generator.py", line 432, in _handle_text super(BytesGenerator,self)._handle_text(msg) File "/usr/lib/python3.7/email/generator.py", line 249, in _handle_text self._write_lines(payload) File "/usr/lib/python3.7/email/generator.py", line 155, in _write_lines self.write(line) File "/usr/lib/python3.7/email/generator.py", line 406, in write self._fp.write(s.encode('ascii', 'surrogateescape')) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 109: ordinal not in range(128)

For the second, there are many content filtering settings not (yet) exposed in Postorius. These include filter_extensions, filter_types, pass_extensions and pass_types which control which MIME parts are removed. For a newly created Mailman 3 list, these four settings are empty, but importing a Mailman 2.1 list will set them according to the corresponding MM 2.1 list's settings.

...

All I found was "convert HTML to plaintext", but does this setting remove attachements?

It would only affect HTML parts, not pdfs.

...

Is there any generel documentation about attachement processing in MM3?

<https://mailman.readthedocs.io/en/latest/src/mailman/handlers/docs/filtering.html>.

...

I would like this mailinglist to accept mails with attachements, send it to the members and archvie the attachments in HyperKitty, to.

You can set Settings -> Alter Messages -> Filter content to No in Postorius to not do any content filtering at all. This will override the Collapse alternatives and Convert html to plaintext settings and pass the message without removing any MIME parts.

But, I think you must have other issues, either with this mail in particular or in general as evidenced by the mail not being received by list members.

Ok, first fix the ascii encode error for me. Any idea how?

thank you.

MfG, Lars Schimmer

TU Graz, Institut für ComputerGraphik & WissensVisualisierung Tel: +43 316 873-5405 E-Mail: l.schimmer@cgv.tugraz.at Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723

On 2/4/20 9:02 AM, Mark Sapiro wrote:

You can fix it by patching the Python library email/generator.py as at <https://github.com/python/cpython/pull/18056/files#diff-220640ea02a24dfdbb0647ef64f1a49b>.

And once you have patched email/generator.py and restarted Mailman core, you can deliver the messages by running the mailman unshunt command, but first look in var/queue/shunt/ to ensure you won't be unshunting any unwanted messages. You can examine those files with mailman qfile.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

Mark Sapiro wrote:

...

This is https://bugs.python.org/issue39384 You can fix it by patching the Python library email/generator.py as at https://github.com/python/cpython/pull/18056/files#diff-220640ea02a24dfdbb06....

Actually, it's more complicated than that. I have closed both the above issue and Pull Request. The underlying issue is https://gitlab.com/mailman/mailman/issues/677 and a fix for that is at https://gitlab.com/mailman/mailman/-/merge_requests/600

-- Mark Sapiro mark at msapiro net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan