Ok so after pulling it all apart this appears to have been a major problem in mailman3 from conception maybe? So here’s what happening as I understand it.
If an email comes in and it is address to 2+ lists and 2 or more of those lists hold the email for moderation. The email is stored in the database using the message-id-hash (or something based on the message-id). Then if one list moderator approve/discards the email then that action deletes the message from the database. But there are still 1+ other lists that the message was addressed to that have pending actions, but that email is now gone. Before 3.3.5 these approves were silently dropped and no email was sent to the list, but in 3.3.5 new code was added to send a dummy email that contains "The content of this message was lost. It was probably cross-posted to multiple lists and previously handled on another list.”
We have multiple campuses in the same city and it’s not uncommon for people to send the same email to the multiple lists in the same city. This is where we’ve started to see the content was lost emails. I’m amazed we haven’t had people report missing emails before this.
My very ugly hack that I’m sure is not even close to the right way of fixing this was to set exim to use max_rcpt = 1 when talking to the mailman LMTP server. Then I made the following changes to runners/lmtp.py
*** lmtp.py.orig 2021-11-04 17:46:03.428874115 +1000 --- /opt/mailman/venv/lib/python3.6/site-packages/mailman/runners/lmtp.py 2021-11-04 17:37:59.911514738 +1000
*** 37,42 **** --- 37,43 ---- import email import asyncio import logging
import hashlib
from aiosmtpd.controller import Controller from aiosmtpd.lmtp import LMTP
*** 155,160 **** --- 156,171 ---- if msg.defects: return ERR_501 msg.original_size = len(envelope.content) +
# Modify the messageID so tha it's unique for each list. This requies# the MTA to be configured to only send one RCPT per connection# In exim this is max_rcpt = 1# SIMON HACKdel msg['Message-ID']msg['Message-ID'] = "{}{}".format(hashlib.sha512(envelope.rcpt_tos[0].encode('utf8')).hexdigest()[:24], message_id.strip("<>"))message_id = msg['Message-ID']# END SIMON HACKadd_message_hash(msg) msg['X-MailFrom'] = envelope.mail_from # RFC 2033 requires us to return a status code for every recipient.
All I’m doing is literally appending a subset of a hash of the mailing list rcpt address to the front of the message-id, so that if it gets moderated, each instance of the email has a different message-id and each moderation action has it’s own copy of the email to operate on. This seems to have gotten us out of the immediate issues. But I’m eager to find a better solution that might be more permanent.
-Simon
Mr Simon Coggins Principal Systems Engineer | Digital Services Directorate CQUniversity Australia, Level 6.14, 160 Ann Street, Brisbane, QLD 4000 P +61 7 3295 1182 | X 51182 | M +61 408 115 861 | E s.coggins@cqu.edu.au<mailto:s.coggins@cqu.edu.au>
[https://www.cqu.edu.au/social-media] I respectfully acknowledge the Traditional Owners of the land on which we work and learn, and pay respect to the First Nations Peoples and their Elders, past, present and future. This communication may contain privileged or confidential information. If you have received this in error, please return to sender and delete. CRICOS: 00219C | RTO Code 40939
On 4 Nov 2021, at 1:11 pm, Simon Coggins <s.coggins@cqu.edu.au<mailto:s.coggins@cqu.edu.au>> wrote:
Sorry that should be we went from 3.3.4 -> 3.3.5. I’m just reading the code now but my quick reading of it seems to say that if someone sees to 2 lists and they are both moderated, if one list moderator releases it, the email is deleted from the store and then when the second person releases it it’s now a dummy message that has no contents? That doesn’t sounds right.
-Simon
Mr Simon Coggins Principal Systems Engineer | Digital Services Directorate CQUniversity Australia, Level 6.14, 160 Ann Street, Brisbane, QLD 4000 P +61 7 3295 1182 | X 51182 | M +61 408 115 861 | E s.coggins@cqu.edu.au<mailto:s.coggins@cqu.edu.au>
[https://www.cqu.edu.au/social-media] I respectfully acknowledge the Traditional Owners of the land on which we work and learn, and pay respect to the First Nations Peoples and their Elders, past, present and future. This communication may contain privileged or confidential information. If you have received this in error, please return to sender and delete. CRICOS: 00219C | RTO Code 40939
On 4 Nov 2021, at 12:50 pm, Simon Coggins <s.coggins@cqu.edu.au<mailto:s.coggins@cqu.edu.au>> wrote:
Since upgrading from 3.3.4 to 3.3.6 we are seeing a high instance of emails that are sent to 2 lists (EG 2 campuses). One campus will get the email and the other campus will get an email that just contains "The content of this message was lost. It was probably cross-posted to multiple lists and previously handled on another list.” Is this something new in 3.3.6? I’m trying to work out what’s going on and why it would be lost. Both lists need the email as the membership list is different so the people on the second list get a useless email.
-Simon
Mr Simon Coggins Principal Systems Engineer | Digital Services Directorate CQUniversity Australia, Level 6.14, 160 Ann Street, Brisbane, QLD 4000 P +61 7 3295 1182 | X 51182 | M +61 408 115 861 | E s.coggins@cqu.edu.au<mailto:s.coggins@cqu.edu.au>
[https://www.cqu.edu.au/social-media] I respectfully acknowledge the Traditional Owners of the land on which we work and learn, and pay respect to the First Nations Peoples and their Elders, past, present and future. This communication may contain privileged or confidential information. If you have received this in error, please return to sender and delete. CRICOS: 00219C | RTO Code 40939