Setting for archiving multi email domains?
Hi again,
I have a Debian-postfix-postgres-nginx-Hyperkitty-Postorius setup with a primary domain (e.g. primarydomain.com) and several 'secondary' domains used for email lists (e.g. emaillistA.com and emaillistB.com). When sending emails to a list using the primarydomain.com address, the email is received, sent to subscribers and archived properly. But when it is sent to one of the secondary domains like tes@emaillistA.com - the email doesn't seem to be sent and isn't archived - despite it being set to be.
What I've tried so far (after searching these archives and looking at the documentation) is to:
- Adding emaillistA.com and emaillistB.com to the MAILMAN_ARCHIVER_FROM setting in the settings_local.py file (and then restart mailman)
- Adding emaillistA.com and emaillistB.com to the mydestination setting in the postfix main.cf file (and the restarting postfix).
...I even tried adding to /etc/hosts 127.0.0.1 emaillistA.com and 127.0.0.1 emaillistB.com but removed it as it didn't seem appropriate there.
Does anyone have any hits of what else I need to configure to get it working for these 'secondary' email domains?
Cheers,
Duane
FYI: in the mailman log was also the following traceback
Mar 04 14:08:24 2021 (5044) Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 84, in process for member in mlist.regular_members.members File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 83, in <genexpr> recipients = set(member.address.email File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 239, in members yield from self._get_members(DeliveryMode.regular) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 226, in _get_members if member.delivery_mode in delivery_modes: File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 212, in delivery_mode return self._lookup('delivery_mode') File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 174, in _lookup pref = getattr(self.address.preferences, preference) AttributeError: 'NoneType' object has no attribute 'preferences' Mar 04 14:08:24 2021 (5044) SHUNTING: 1614866904.2741017+f1f320a5d0006e885586be5b33543e364e3829b4
On 3/4/21 6:30 AM, Duane Raymond wrote:
FYI: in the mailman log was also the following traceback
Mar 04 14:08:24 2021 (5044) Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 84, in process for member in mlist.regular_members.members File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 83, in <genexpr> recipients = set(member.address.email File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 239, in members yield from self._get_members(DeliveryMode.regular) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 226, in _get_members if member.delivery_mode in delivery_modes: File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 212, in delivery_mode return self._lookup('delivery_mode') File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 174, in _lookup pref = getattr(self.address.preferences, preference) AttributeError: 'NoneType' object has no attribute 'preferences' Mar 04 14:08:24 2021 (5044) SHUNTING: 1614866904.2741017+f1f320a5d0006e885586be5b33543e364e3829b4
Assuming this relates to the missing messages, there is no issue with delivery to Mailman. This message is getting to Mailman but throws the above exception during processing and is shunted.
You can examine this messages with mailman qfile var/queue/shunt/1614866904.2741017+f1f320a5d0006e885586be5b33543e364e3829b4.pck
and similarly for other shunted messages.
The issue here is Mailman is going through the roster of regular (non-digest) list members to find those with delivery enabled and it finds a member that has no associated address. Can you view the members of this list?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Duane Raymond writes:
I have a Debian-postfix-postgres-nginx-Hyperkitty-Postorius setup with a primary domain (e.g. primarydomain.com) and several 'secondary' domains used for email lists (e.g. emaillistA.com and emaillistB.com). When sending emails to a list using the primarydomain.com address, the email is received, sent to subscribers and archived properly. But when it is sent to one of the secondary domains like tes@emaillistA.com - the email doesn't seem to be sent and isn't archived - despite it being set to be.
What do you mean by "set"? That the list has archiving enabled?
What is in the Postfix logs? Where does Postfix try to deliver the second class of email (list-specific domains)? Is there a "bounce" (more precisely, a Delivery Status Notice telling you that the mail was not delivered, and with luck, why Postfix had a problem)? What is in the Mailman logs?
I assume from your description (one server, multiple domains) that these list-specific domains are "virtual" domains that are handled by the single server. It seems likely that something is going wrong with the routing in Postfix.
What I've tried so far (after searching these archives and looking at the documentation) is to:
- Adding emaillistA.com and emaillistB.com to the MAILMAN_ARCHIVER_FROM setting in the settings_local.py file (and then restart mailman)
- Adding emaillistA.com and emaillistB.com to the mydestination setting in the postfix main.cf file (and the restarting postfix).
(2) above will allow receiving mail for those domains, but is unlikely to be sufficient for routing to lists. You also need to inform Mailman about the virtual domains, because the internal name of the mailing list includes the domain.
Postfix has extensive support for virtual domains. This is somewhat tricky. Are you familiar with this support, as documented in http://www.postfix.org/VIRTUAL_README.html, and the general docs? For Mailman's needs from Postfix, see https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht....
If the above aren't sufficient hints, we need more information about both the Postfix configuration and the Mailman configuration.
In a later message https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/..., you report at the end of the traceback (line folded for readability):
AttributeError: 'NoneType' object has no attribute 'preferences' Mar 04 14:08:24 2021 (5044) SHUNTING: 1614866904.2741017+f1f320a5d0006e885586be5b33543e364e3829b4
This seems to indicate that either your list has a member that doesn't exist in Mailman's subscriber database or (maybe) that there are no subscribers, because it's trying to get profile information about a nonexistent list member (represented by the Python 'None' object).
Did this occur only once? How many messages are in the "shunt" queue?
Thanks!
So - I've documented what I did below. Now that I am getting bouces, I got an email that said "The response from the remote server was: 550 5.1.1 <test@emaildomainA.com>: Recipient address rejected: User unknown in virtual alias table"
Specifically to some ot the questions above:
What do you mean by "set"? That the list has archiving enabled?
Yes. I meant that archiving had been enabled for the test@emaildomainA.com test list.
Shunting error: I think the shunting error is a distraction if it is caused by not having a recipient. Early tests to the test list had me as an administrator, but not as a member - so this would have caused that. Once I added myself (before I sent the original message in this thread), the result (not sending the message to the list using a 'virtual' domain and not archiving it) continued.
Virtual Domains: So based on the Postfix virtual domain documentation and the Mailman3 specific "Unusual Postfix configuration' section, I added the following to my postfix main.cf file: virtual_alias_domains = emaillistA.com emaillistB.com virtual_alias_maps = hash:/opt/mailman/mm/var/data/postfix_vmap and removed those domains from the 'mydestination' setting.
I then created the postfix_vmap file and added to it: emaillistA.com DOMAIN # emaillistB.com DOMAIN
...then ran postmap with the path and reloaded postfix.
On testing I also noticed that a previous counce was to emaildomainC.com - an alias for emaildomainA.com I had originally setup in the Postorius domain setup and then removed. I guessed that it was still in the postfix config, so I double checked in the MM generated postfix_domains file and ran postmap /path-to/postfix_domains and reloaded postfix again...and sent another test message....
This then generated the error email at the top from my email client (gmail).
...but the end result is no email sent and thus the reason there is no archive....
Any further suggestions? I can supply other log messages and configuration options if it helps.
Duane
On 3/4/21 12:52 PM, Duane Raymond wrote:
Virtual Domains: So based on the Postfix virtual domain documentation and the Mailman3 specific "Unusual Postfix configuration' section, I added the following to my postfix main.cf file: virtual_alias_domains = emaillistA.com emaillistB.com virtual_alias_maps = hash:/opt/mailman/mm/var/data/postfix_vmap and removed those domains from the 'mydestination' setting.
You probably do not have a configuration that is a "Unusual Postfix configuration"
Putting Mailman aside, is there some reason why emaillistA.com and emaillistB.com need to be virtual_alias_domains? If these domains are only for mail lists, they do not need to be virtual_alias_domains?
I then created the postfix_vmap file and added to it: emaillistA.com DOMAIN # emaillistB.com DOMAIN
You don't create these files. Mailman creates them in its var/data
directory when you run mailman aliases or just start mailman core.
...then ran postmap with the path and reloaded postfix.
Mailman also runs the appropriate postmap commands.
You should have one of two setups in Postfix depending on whether emaillistA.com and emaillistB.com need to be virtual_alias_domains or not. If they are only for list mail, they should not be virtual_alias_domains in Postfix and their domain definitions in Mailman should not have an alias domain. In that case, in Postfix you want
transport_maps = hash:/path-to-mailman/var/data/postfix_lmtp local_recipient_maps = hash:/path-to-mailman/var/data/postfix_lmtp relay_domains = hash:/path-to-mailman/var/data/postfix_domains
and you want Mailman to generate those files. Also see the note in <https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.html#transport-maps> about adding hash:/path-to-mailman/var/data/postfix_lmtp to local_recipient_maps rather than replacing it.
If emaillistA.com and emaillistB.com do need to be virtual_alias_domains, you configure them in postfix according to the normal postfix configurationfor virtual_alias_domains and virtual_alias_maps.
Then each of these domains needs a corresponding, otherwise unused alias domain defined in Mailman and what you need in Postfix is
transport_maps = hash:/path-to-mailman/var/data/postfix_lmtp relay_domains = hash:/path-to-mailman/var/data/postfix_domains virtual_alias_maps = hash:/path-to-mailman/var/data/postfix_vmap
Where again if you have existing settings for these things, add the mailman files rather that replacing the existing with them.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks for the continued input on this. It's really appreciated!
I've reverted the changes I made with virtual domains and tried again (after restarting postfix and mailman). The result back was like
<test@oldlist.com> (expanded from <test@emaillistA.com>): host oldlisthost.com[ip] said: 550 5.1.1 <test@oldlist.com>: Recipient address rejected: User unknown in virtual alias table (in reply to RCPT TO command)
But I can't find oldlist.com in the configuration settings. It was set as an alias for emaillistA.com domain, but was removed.
So still looking for a solution...am considering a reinstall from scratch....but that seems rather drastic!
Duane
On 3/5/21 10:12 AM, Duane Raymond wrote:
Thanks for the continued input on this. It's really appreciated!
I've reverted the changes I made with virtual domains and tried again (after restarting postfix and mailman). The result back was like
<test@oldlist.com> (expanded from <test@emaillistA.com>): host oldlisthost.com[ip] said: 550 5.1.1 <test@oldlist.com>: Recipient address rejected: User unknown in virtual alias table (in reply to RCPT TO command)
But I can't find oldlist.com in the configuration settings. It was set as an alias for emaillistA.com domain, but was removed.
So still looking for a solution...am considering a reinstall from scratch....but that seems rather drastic!
Please post the output from postconf -n
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
fairsayforum.com = mail site domain (working fine for lists, archives and web)
alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases always_add_missing_headers = yes append_dot_mydomain = no biff = no compatibility_level = 2 default_destination_concurrency_limit = 15 default_destination_recipient_limit = 30 delay_warning_time = 1h header_checks = regexp:/etc/postfix/header_checks inet_interfaces = all inet_protocols = all local_recipient_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp mailbox_size_limit = 0 milter_default_action = accept milter_protocol = 2 mydestination = $myhostname, mail.fairsayforum.com, localhost.fairsayforum.com, localhost myhostname = mail.fairsayforum.com mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 myorigin = /etc/mailname non_smtpd_milters = inet:localhost:8892 owner_request_special = no readme_directory = no recipient_delimiter = + relay_domains = hash:/opt/mailman/mm/var/data/postfix_domains relayhost = smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) smtpd_milters = inet:localhost:8892 smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtpd_use_tls = yes transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp unknown_local_recipient_reject_code = 550
On 3/5/21 10:19 AM, Duane Raymond wrote:
fairsayforum.com = mail site domain (working fine for lists, archives and web)
alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases always_add_missing_headers = yes append_dot_mydomain = no biff = no compatibility_level = 2 default_destination_concurrency_limit = 15 default_destination_recipient_limit = 30 delay_warning_time = 1h header_checks = regexp:/etc/postfix/header_checks inet_interfaces = all inet_protocols = all local_recipient_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp mailbox_size_limit = 0 milter_default_action = accept milter_protocol = 2 mydestination = $myhostname, mail.fairsayforum.com, localhost.fairsayforum.com, localhost myhostname = mail.fairsayforum.com mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 myorigin = /etc/mailname non_smtpd_milters = inet:localhost:8892 owner_request_special = no readme_directory = no recipient_delimiter = + relay_domains = hash:/opt/mailman/mm/var/data/postfix_domains relayhost = smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) smtpd_milters = inet:localhost:8892 smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtpd_use_tls = yes transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp unknown_local_recipient_reject_code = 550
And previously
<test@oldlist.com> (expanded from <test@emaillistA.com>): host oldlisthost.com[ip] said: 550 5.1.1 <test@oldlist.com>: Recipient address rejected: User unknown in virtual alias table (in reply to RCPT TO command)
But I can't find oldlist.com in the configuration settings. It was set as an alias for emaillistA.com domain, but was removed.
Have you removed the alias domain from the emaillistA.com domain and
similarly for the others and then regenerated the maps with mailman aliases or mailman start -gafter removing the alias domains.
Also, I'm confused about "User unknown in virtual alias table" since there are no virtual_alias_domains in your Postfix main.cf. Have you reloaded Postfix since updating main.cf?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
And previously (in response to the postconf -n result)
I didn't keep a record of each edit of the main.cf file. I did try things like adding emailistA.com to the mydestination list (and localhost.emailistA.com like for the maindomain.com), but nothing seemed to change.
There was also a value for milter_default_action which has been commented out as apparently it was added by opendkim.
Have you removed the alias domain from the emaillistA.com domain and similarly for the others and then regenerated the maps with mailman aliases or mailman start -g after removing the alias domains.
I've done that now and but it doesn't seem to have changed anything. At present emails to the emaillistA.com domain (vs the maindomain.com) are being shunted again.
Also, I'm confused about "User unknown in virtual alias table" since there are no virtual_alias_domains in your Postfix main.cf. Have you reloaded Postfix since updating main.cf?
I did again and this bounce message has disappeared - but instead it is shunting - so no bounce comes back now.
I'll go through the documentation again line by line and see if there is something I missed.
On 3/8/21 3:38 AM, Duane Raymond wrote:
I did again and this bounce message has disappeared - but instead it is shunting - so no bounce comes back now.
If messages are shunted, they are being delivered to Mailman, so the Postfix issue is apparently resolved.
What is the full error including traceback from Mailman's mailman.log for the shunted messages?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
What is the full error including traceback from Mailman's mailman.log for the shunted messages?
My test just now raised the following error:
Mar 09 20:52:55 2021 (26267) ACCEPT: <CAMs==2HNnd_6+4=iWZhK3d7oTvqk2W80wMdCJ+Y9eRFUKno1mA@mail.gmail.com> Mar 09 20:52:56 2021 (26271) Uncaught runner exception: 'NoneType' object has no attribute 'preferences' Mar 09 20:52:56 2021 (26271) Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 84, in process for member in mlist.regular_members.members File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 83, in <genexpr> recipients = set(member.address.email File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 239, in members yield from self._get_members(DeliveryMode.regular) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 226, in _get_members if member.delivery_mode in delivery_modes: File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 212, in delivery_mode return self._lookup('delivery_mode') File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 174, in _lookup pref = getattr(self.address.preferences, preference) AttributeError: 'NoneType' object has no attribute 'preferences' Mar 09 20:52:56 2021 (26271) SHUNTING: 1615323176.8137882+cd8501dd322d0ca6b4a3a187d57bc691cd0ec343
and running the following command raised the result below root@mail:~# mailman qfile var/queue/shunt/1615323176.8137882+cd8501dd322d0ca6b4a3a187d57bc691cd0ec343.pck
[----- start pickle -----] <----- start object 1 -----> Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by mail.fairsayforum.com (Postfix) with ESMTPS id DAB464236D for <test@campaigningforum.com>; Tue, 9 Mar 2021 20:52:54 +0000 (UTC) Received: by mail-ej1-x62e.google.com with SMTP id mm21so31549207ejb.12 for <test@campaigningforum.com>; Tue, 09 Mar 2021 12:52:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fairsay.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=EgptMSUIn7D50wRC/g0b3CRiIpplElahV/29ypgLKLk=; b=kw8eSDmuxOf1yjek6lbb1wMr7gG/F/3mteeWDjd0xka58taUMLjyI97iXivysixDJv PuUL24pBi4VpIhZat19rC0rVP8jeAbUMOVDBrQdJ/9comxSsxWTeqnvxXyoSfPtJD4sw U0ytrse1sOGme5OXWQ433+10i9b/pUN0umYe8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=EgptMSUIn7D50wRC/g0b3CRiIpplElahV/29ypgLKLk=; b=mOnXvqUQj9zgzoFLUkgEKDyuyH8WdyYA/KhYelsnqXITSXdoMsxw9yHneoUkZNh+96 ZjONXBkZhlDvmonwINQKF2sLOQ6RRoAdKnH16ecmc7rP+1zT9nGXFkYvNQMcFHCahD8T PPVgjmEuBkOC+6qAWE5oIWY7qehYw1sQtQdqlk2E4qgyPGrWen17NoMP7J0dlHaMprL8 +M26NpELA35CyLOfJuuLq4lBNUb1jq6rNlFkpJTuPGNJror6YSDxeB5jQR8XlErr+G8S Wfwgq98pf4+aPHR8CYGOjVAtFaITZWgjZPDC+IeAmICsu04V4Vn6d/p4hUeGLnQF39le dLlw== X-Gm-Message-State: AOAM532vFY+zWnSLvJnCElbuzUooTINttm38iDprNZmsxMc/DRls5JrB q17o0OwrGirg6bKKbkaskSLkQ/K4xnaPVorxc9zZkd0eiU27+sS8 X-Google-Smtp-Source: ABdhPJxC/QTIs31D7ELVUIIF4tE5go1LieVZjr+GrWo2/12dRVznfTMEJsgbMJLjdjTBzFjxYK33Xm3QgNd2L4suX+s= X-Received: by 2002:a17:906:1a16:: with SMTP id i22mr5432371ejf.522.1615323174025; Tue, 09 Mar 2021 12:52:54 -0800 (PST) MIME-Version: 1.0 From: Duane Raymond <duane.raymond@fairsay.com> Date: Tue, 9 Mar 2021 21:52:36 +0100 Message-ID: <CAMs==2HNnd_6+4=iWZhK3d7oTvqk2W80wMdCJ+Y9eRFUKno1mA@mail.gmail.com> Subject: Test to identify shunting error To: test@campaigningforum.com Content-Type: multipart/alternative; boundary="000000000000fc503e05bd20bba9" Message-ID-Hash: 3THPVXHXTHRNBXZHN2TD5VBUWIMXOEY2 X-Message-ID-Hash: 3THPVXHXTHRNBXZHN2TD5VBUWIMXOEY2 X-MailFrom: duane.raymond@fairsay.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
--000000000000fc503e05bd20bba9 Content-Type: text/plain; charset="UTF-8"
Test to identify shunting error
--000000000000fc503e05bd20bba9 Content-Type: text/html; charset="UTF-8"
<div dir="ltr">Test to identify shunting error<br></div>
--000000000000fc503e05bd20bba9--
<----- start object 2 -----> { '_parsemsg': False, 'envsender': 'noreply@campaigningforum.com', 'lang': 'en', 'listid': 'test.campaigningforum.com', 'original_size': 2436, 'received_time': datetime.datetime(2021, 3, 9, 20, 52, 54, 941548), 'rule_hits': [], 'rule_misses': [ 'dmarc-mitigation', 'no-senders', 'approved', 'emergency', 'loop', 'banned-address', 'member-moderation', 'nonmember-moderation', 'administrivia', 'implicit-dest', 'max-recipients', 'max-size', 'news-moderation', 'no-subject', 'digests', 'suspicious-header'], 'to_list': True, 'version': 3, 'whichq': 'pipeline'} [----- end pickle -----]
On 3/9/21 12:59 PM, Duane Raymond wrote:
What is the full error including traceback from Mailman's mailman.log for the shunted messages?
My test just now raised the following error:
...> File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 174, in _lookup
pref = getattr(self.address.preferences, preference)AttributeError: 'NoneType' object has no attribute 'preferences' Mar 09 20:52:56 2021 (26271) SHUNTING: 1615323176.8137882+cd8501dd322d0ca6b4a3a187d57bc691cd0ec343
and running the following command raised the result below root@mail:~# mailman qfile var/queue/shunt/1615323176.8137882+cd8501dd322d0ca6b4a3a187d57bc691cd0ec343.pck
That result is as expected.
It isn't the message that causes the problem. We are now back to <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/EYKOCRMHTPGYE6UT5EZGQGFGTDUQC55L/>
As I said there, Mailman is going through the roster of regular (non-digest) list members to find those with delivery enabled and it finds a member that has no associated address. Can you view the members of this list?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
As I said there, Mailman is going through the roster of regular (non-digest) list members to find those with delivery enabled and it finds a member that has no associated address. Can you view the members of this list?
Yes - the list has three members (test list): me and two others. For my subscription I have gone through the membership options and explicitly set them. For the others their membership options are all unselected and thus I assume defaulting to a fallback value. (not sure if that matters)
On 3/9/21 4:05 PM, Duane Raymond wrote:
As I said there, Mailman is going through the roster of regular (non-digest) list members to find those with delivery enabled and it finds a member that has no associated address. Can you view the members of this list?
Yes - the list has three members (test list): me and two others. For my subscription I have gone through the membership options and explicitly set them. For the others their membership options are all unselected and thus I assume defaulting to a fallback value. (not sure if that matters)
What matters is at least one of those members doesn't have an associated address record.
If you have access to mailman shell do the following:
mailman shell -l LIST_ID Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the LIST_ID mailing list
for mbr in m.members.members: ... if mbr.address is None: ... print('user: {}, user.addresses: {}'.format( ''' mbr.user, mbr.user.addresses)) ...
And see what that gives you. (At the last ... prompt just type enter)
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
I think I have it working thanks to some hints in your questions. here is the process I took in case others experience it the same or in case anyone wishes to comment!
When I use the mailman shell comment it seems to reflect just what you posted (I also tried test.campaigningforum.com and got the same result): (venv) mailman@mail:/opt/mailman/mm$ mailman shell -l test@campaigningforum.com Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the test@campaigningforum.com mailing list
Was there something beyond running this and getting the output I was supposed to enter? Was an error expected if one members doesn't have an associated address record?
The membership list is simply three emails persona@domaina.com persona_surname@domaina.com "Duane Raymond" <duane.raymond@fairsay.com>
There was one non-member address ("Duane Raymond" <duane.raymond@fairsay.com> - notice it is the same as the member address above) which I removed and the same member address above stayed listed.
I then sent another test email and still nothing was distributed or archived.
After searching around for a cause of the error message, I found this: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/V... which mentioning missing ros/values in the database tables... so then I took a different route.
I accessed the database tables directly to see if there was anything strange that related to your comment that "at least one of those members doesn't have an associated address record." What I found in the 'members' table were two rows which were not listed in the user listing of the mailing list GUI. So I deleted these and tried again.
This time the posting was archived and I did get an acknowledgement of the post, but despite having the option "Receive own postings" set to "yes", I didn't receive it. So I continued to investigate the database table and found that between the tables 'member' -> user and member -> address, the member table had the 'address_id' value but a null 'user_id' - so I fixed these and tried again. So far, there is still no email sent but archiving works fine and there is an acknowledgement email.
In the log files mail.info mail.log it seems it was sent as the message is: Mar 10 14:17:57 mail postfix/smtp[17747]: 844B74236D: to=<duane.raymond@fairsay.com>, relay=aspmx.l.google.com[2a00:1450:4013:c07::1a]:25, delay=0.6, delays=0.01/0/0.42/0.18, dsn=2.0.0, status=sent (250 2.0.0 OK 1615385877 t8si7488737ejj.661 - gsmtp) Mar 10 14:17:57 mail postfix/qmgr[4201]: 844B74236D: removed
So - in conclusion - the main problem now (after fixing the relationships in the tables which solved the archiving problem) seems to be that emails are not being received (and thus in a queue somewhere?) since the join email to the test list is sent and received immediately.
...sorry for the long post...but wanted to document what I've been trying and finding!
Duane
Update: the other person on the test list has indicated they are receiving the emails now - and that gmail may be the cause of the issue by filtering out copies of the message I had sent. (my fairsay.com address is gmail hosted, but another is not and is simply forwarded there).
...so perhaps the problem is solved (will do a little more testing to verify!)
Duane
On 3/10/21 9:49 AM, Duane Raymond wrote:
Update: the other person on the test list has indicated they are receiving the emails now - and that gmail may be the cause of the issue by filtering out copies of the message I had sent. (my fairsay.com address is gmail hosted, but another is not and is simply forwarded there).
...so perhaps the problem is solved (will do a little more testing to verify!)
It's Gmail. Just check the mail.log and look for status=sent
-- Brian Carpenter Harmonylists.com Emwd.com
On Wed, 10 Mar 2021 at 15:52, Brian Carpenter <brian_carpenter@emwd.com> wrote:
Update: the other person on the test list has indicated they are receiving the emails now - and that gmail may be the cause of the issue by filtering out copies of the message I had sent. (my fairsay.com address is gmail hosted, but another is not and is simply forwarded there).
...so perhaps the problem is solved (will do a little more testing to verify!)
It's Gmail. Just check the mail.log and look for status=sent
Thanks Brian - then I think it is fixed :-) Thank you Mark and Brian for your patience and persistence. I'll keep monitoring the situation as new subscribers (not sure if all) seem to also be missing the user id in the members table and I then need to run an update query to fix it - but at least I know what to look for and how to resolve it! (this last week was painful!)
Cheers,
Duane
On 3/10/21 6:37 AM, Duane Raymond wrote:
Hi Mark,
I think I have it working thanks to some hints in your questions. here is the process I took in case others experience it the same or in case anyone wishes to comment!
When I use the mailman shell comment it seems to reflect just what you posted (I also tried test.campaigningforum.com and got the same result): (venv) mailman@mail:/opt/mailman/mm$ mailman shell -l test@campaigningforum.com Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the test@campaigningforum.com mailing list
Was there something beyond running this and getting the output I was supposed to enter? Was an error expected if one members doesn't have an associated address record?
Did you enter this following the initial >>> prompt?
for mbr in m.members.members: ... if mbr.address is None: ... print('user: {}, user.addresses: {}'.format( ''' mbr.user, mbr.user.addresses)) ...
If so, it was expected to print the representation of the user object and that objects addresses list fo any member whose address attribute is None.
...
This time the posting was archived and I did get an acknowledgement of the post, but despite having the option "Receive own postings" set to "yes", I didn't receive it.
As you have determined, this is a googlemail issue. See <https://wiki.list.org/x/4030680>.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Hi Mark,
The loop you mentioned I missed actually (and in the process may have found a bug in hyperkitty).
After running the code in the mailman shell:
for mbr in m.members.members: if mbr.address is None: print('user: {}, user.addresses: {}'.format(mbr.user, mbr.user.addresses))
I didn't get any result in the test list, presumably because I had fixed that in the db directly. However on running it on the main migrated list, I found two entries with the problem - perhaps not surprisingly exactly the same member as had caused the problem with the test list. Once I fixed it (adding a preferred address id to the user table), the result was no more entries were returned. I also tried it on another migrated list and it came 'clean' as well.
As for the potential bug in hyperkitty, I have been referencing the hyperkitty version of this thread as I was working the problem, and while I faintly remember seeing the loop in the email on my phone, when I was referencing the hyperkitty version of your email it was not there - even when expanding the quoted text....perhaps you see the same thing via hyperkitty? Essentially it removed part of the message that was intended to be seen.
Regarding the email duplicate issue - thanks for pointing me to that posting - glad to know it isn't just me!
Cheers,
Duane
On 3/10/21 2:32 PM, Duane Raymond wrote:
As for the potential bug in hyperkitty, I have been referencing the hyperkitty version of this thread as I was working the problem, and while I faintly remember seeing the loop in the email on my phone, when I was referencing the hyperkitty version of your email it was not there - even when expanding the quoted text....perhaps you see the same thing via hyperkitty? Essentially it removed part of the message that was intended to be seen.
That's a known issue with the HyperKitty that is installed at mailman3.org and python.org due to the patch for markdown rendering <https://gitlab.com/mailman/hyperkitty/-/merge_requests/160>. This patch is superceded by <https://gitlab.com/mailman/hyperkitty/-/merge_requests/324>, but I haven't gotten around to trying that yet.
If you look at the message view at <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/ZZIAHLHDSJJ3AJEZ52VH3TOAE5XU6IDA/>, you'll see the missing part
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 3/10/21 3:46 PM, Mark Sapiro wrote:
That's a known issue with the HyperKitty that is installed at mailman3.org and python.org due to the patch for markdown rendering <https://gitlab.com/mailman/hyperkitty/-/merge_requests/160>. This patch is superceded by <https://gitlab.com/mailman/hyperkitty/-/merge_requests/324>, but I haven't gotten around to trying that yet.
I just installed the changes from <https://gitlab.com/mailman/hyperkitty/-/merge_requests/324> on lists.mailman3.org and the thread view is now better.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I just installed the changes from https://gitlab.com/mailman/hyperkitty/-/merge_requests/324 on lists.mailman3.org and the thread view is now better.
Fantastic! Now it is readable via hyperkitty :-) (not 'perfect' but then I know parsing unstructured text perfectly when you don't know what it will contain is very difficult!)
participants (4)
-
Brian Carpenter -
Duane Raymond -
Mark Sapiro -
Stephen J. Turnbull