Duane Raymond writes:

I have a Debian-postfix-postgres-nginx-Hyperkitty-Postorius setup with a primary domain (e.g. primarydomain.com) and several 'secondary' domains used for email lists (e.g. emaillistA.com and emaillistB.com). When sending emails to a list using the primarydomain.com address, the email is received, sent to subscribers and archived properly. But when it is sent to one of the secondary domains like tes@emaillistA.com - the email doesn't seem to be sent and isn't archived - despite it being set to be.

What do you mean by "set"? That the list has archiving enabled?

What is in the Postfix logs? Where does Postfix try to deliver the second class of email (list-specific domains)? Is there a "bounce" (more precisely, a Delivery Status Notice telling you that the mail was not delivered, and with luck, why Postfix had a problem)? What is in the Mailman logs?

I assume from your description (one server, multiple domains) that these list-specific domains are "virtual" domains that are handled by the single server. It seems likely that something is going wrong with the routing in Postfix.

What I've tried so far (after searching these archives and looking at the documentation) is to:

1. Adding emaillistA.com and emaillistB.com to the MAILMAN_ARCHIVER_FROM setting in the settings_local.py file (and then restart mailman)
2. Adding emaillistA.com and emaillistB.com to the mydestination setting in the postfix main.cf file (and the restarting postfix).

(2) above will allow receiving mail for those domains, but is unlikely to be sufficient for routing to lists. You also need to inform Mailman about the virtual domains, because the internal name of the mailing list includes the domain.

Postfix has extensive support for virtual domains. This is somewhat tricky. Are you familiar with this support, as documented in http://www.postfix.org/VIRTUAL_README.html, and the general docs? For Mailman's needs from Postfix, see https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/docs/mta.ht....

If the above aren't sufficient hints, we need more information about both the Postfix configuration and the Mailman configuration.

In a later message https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/..., you report at the end of the traceback (line folded for readability):

AttributeError: 'NoneType' object has no attribute 'preferences' Mar 04 14:08:24 2021 (5044) SHUNTING: 1614866904.2741017+f1f320a5d0006e885586be5b33543e364e3829b4

This seems to indicate that either your list has a member that doesn't exist in Mailman's subscriber database or (maybe) that there are no subscribers, because it's trying to get profile information about a nonexistent list member (represented by the Python 'None' object).

Did this occur only once? How many messages are in the "shunt" queue?

Thanks!

So - I've documented what I did below. Now that I am getting bouces, I got an email that said "The response from the remote server was: 550 5.1.1 <test@emaildomainA.com>: Recipient address rejected: User unknown in virtual alias table"

Specifically to some ot the questions above:

What do you mean by "set"? That the list has archiving enabled?

Yes. I meant that archiving had been enabled for the test@emaildomainA.com test list.

Shunting error: I think the shunting error is a distraction if it is caused by not having a recipient. Early tests to the test list had me as an administrator, but not as a member - so this would have caused that. Once I added myself (before I sent the original message in this thread), the result (not sending the message to the list using a 'virtual' domain and not archiving it) continued.

Virtual Domains: So based on the Postfix virtual domain documentation and the Mailman3 specific "Unusual Postfix configuration' section, I added the following to my postfix main.cf file: virtual_alias_domains = emaillistA.com emaillistB.com virtual_alias_maps = hash:/opt/mailman/mm/var/data/postfix_vmap and removed those domains from the 'mydestination' setting.

I then created the postfix_vmap file and added to it: emaillistA.com DOMAIN # emaillistB.com DOMAIN

...then ran postmap with the path and reloaded postfix.

On testing I also noticed that a previous counce was to emaildomainC.com - an alias for emaildomainA.com I had originally setup in the Postorius domain setup and then removed. I guessed that it was still in the postfix config, so I double checked in the MM generated postfix_domains file and ran postmap /path-to/postfix_domains and reloaded postfix again...and sent another test message....

This then generated the error email at the top from my email client (gmail).

...but the end result is no email sent and thus the reason there is no archive....

Any further suggestions? I can supply other log messages and configuration options if it helps.

Duane

Thanks for the continued input on this. It's really appreciated!

I've reverted the changes I made with virtual domains and tried again (after restarting postfix and mailman). The result back was like

<test@oldlist.com> (expanded from <test@emaillistA.com>): host oldlisthost.com[ip] said: 550 5.1.1 <test@oldlist.com>: Recipient address rejected: User unknown in virtual alias table (in reply to RCPT TO command)

But I can't find oldlist.com in the configuration settings. It was set as an alias for emaillistA.com domain, but was removed.

So still looking for a solution...am considering a reinstall from scratch....but that seems rather drastic!

Duane

fairsayforum.com = mail site domain (working fine for lists, archives and web)

alias_database = hash:/etc/aliases alias_maps = hash:/etc/aliases always_add_missing_headers = yes append_dot_mydomain = no biff = no compatibility_level = 2 default_destination_concurrency_limit = 15 default_destination_recipient_limit = 30 delay_warning_time = 1h header_checks = regexp:/etc/postfix/header_checks inet_interfaces = all inet_protocols = all local_recipient_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp mailbox_size_limit = 0 milter_default_action = accept milter_protocol = 2 mydestination = $myhostname, mail.fairsayforum.com, localhost.fairsayforum.com, localhost myhostname = mail.fairsayforum.com mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 myorigin = /etc/mailname non_smtpd_milters = inet:localhost:8892 owner_request_special = no readme_directory = no recipient_delimiter = + relay_domains = hash:/opt/mailman/mm/var/data/postfix_domains relayhost = smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) smtpd_milters = inet:localhost:8892 smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination smtpd_tls_cert_file = /etc/ssl/certs/ssl-cert-snakeoil.pem smtpd_tls_key_file = /etc/ssl/private/ssl-cert-snakeoil.key smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache smtpd_use_tls = yes transport_maps = hash:/opt/mailman/mm/var/data/postfix_lmtp unknown_local_recipient_reject_code = 550

And previously (in response to the postconf -n result)

I didn't keep a record of each edit of the main.cf file. I did try things like adding emailistA.com to the mydestination list (and localhost.emailistA.com like for the maindomain.com), but nothing seemed to change.

There was also a value for milter_default_action which has been commented out as apparently it was added by opendkim.

Have you removed the alias domain from the emaillistA.com domain and similarly for the others and then regenerated the maps with mailman aliases or mailman start -g after removing the alias domains.

I've done that now and but it doesn't seem to have changed anything. At present emails to the emaillistA.com domain (vs the maindomain.com) are being shunted again.

Also, I'm confused about "User unknown in virtual alias table" since there are no virtual_alias_domains in your Postfix main.cf. Have you reloaded Postfix since updating main.cf?

I did again and this bounce message has disappeared - but instead it is shunting - so no bounce comes back now.

I'll go through the documentation again line by line and see if there is something I missed.

What is the full error including traceback from Mailman's mailman.log for the shunted messages?

My test just now raised the following error:

Mar 09 20:52:55 2021 (26267) ACCEPT: <CAMs==2HNnd_6+4=iWZhK3d7oTvqk2W80wMdCJ+Y9eRFUKno1mA@mail.gmail.com> Mar 09 20:52:56 2021 (26271) Uncaught runner exception: 'NoneType' object has no attribute 'preferences' Mar 09 20:52:56 2021 (26271) Traceback (most recent call last): File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 173, in _one_iteration self._process_one_file(msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/runner.py", line 266, in _process_one_file keepqueued = self._dispose(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/runners/pipeline.py", line 37, in _dispose process(mlist, msg, msgdata, pipeline) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/core/pipelines.py", line 50, in process handler.process(mlist, msg, msgdata) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 84, in process for member in mlist.regular_members.members File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/handlers/member_recipients.py", line 83, in <genexpr> recipients = set(member.address.email File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 239, in members yield from self._get_members(DeliveryMode.regular) File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/roster.py", line 226, in _get_members if member.delivery_mode in delivery_modes: File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 212, in delivery_mode return self._lookup('delivery_mode') File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailman/model/member.py", line 174, in _lookup pref = getattr(self.address.preferences, preference) AttributeError: 'NoneType' object has no attribute 'preferences' Mar 09 20:52:56 2021 (26271) SHUNTING: 1615323176.8137882+cd8501dd322d0ca6b4a3a187d57bc691cd0ec343

and running the following command raised the result below root@mail:~# mailman qfile var/queue/shunt/1615323176.8137882+cd8501dd322d0ca6b4a3a187d57bc691cd0ec343.pck

[----- start pickle -----] <----- start object 1 -----> Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by mail.fairsayforum.com (Postfix) with ESMTPS id DAB464236D for <test@campaigningforum.com>; Tue, 9 Mar 2021 20:52:54 +0000 (UTC) Received: by mail-ej1-x62e.google.com with SMTP id mm21so31549207ejb.12 for <test@campaigningforum.com>; Tue, 09 Mar 2021 12:52:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fairsay.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=EgptMSUIn7D50wRC/g0b3CRiIpplElahV/29ypgLKLk=; b=kw8eSDmuxOf1yjek6lbb1wMr7gG/F/3mteeWDjd0xka58taUMLjyI97iXivysixDJv PuUL24pBi4VpIhZat19rC0rVP8jeAbUMOVDBrQdJ/9comxSsxWTeqnvxXyoSfPtJD4sw U0ytrse1sOGme5OXWQ433+10i9b/pUN0umYe8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=EgptMSUIn7D50wRC/g0b3CRiIpplElahV/29ypgLKLk=; b=mOnXvqUQj9zgzoFLUkgEKDyuyH8WdyYA/KhYelsnqXITSXdoMsxw9yHneoUkZNh+96 ZjONXBkZhlDvmonwINQKF2sLOQ6RRoAdKnH16ecmc7rP+1zT9nGXFkYvNQMcFHCahD8T PPVgjmEuBkOC+6qAWE5oIWY7qehYw1sQtQdqlk2E4qgyPGrWen17NoMP7J0dlHaMprL8 +M26NpELA35CyLOfJuuLq4lBNUb1jq6rNlFkpJTuPGNJror6YSDxeB5jQR8XlErr+G8S Wfwgq98pf4+aPHR8CYGOjVAtFaITZWgjZPDC+IeAmICsu04V4Vn6d/p4hUeGLnQF39le dLlw== X-Gm-Message-State: AOAM532vFY+zWnSLvJnCElbuzUooTINttm38iDprNZmsxMc/DRls5JrB q17o0OwrGirg6bKKbkaskSLkQ/K4xnaPVorxc9zZkd0eiU27+sS8 X-Google-Smtp-Source: ABdhPJxC/QTIs31D7ELVUIIF4tE5go1LieVZjr+GrWo2/12dRVznfTMEJsgbMJLjdjTBzFjxYK33Xm3QgNd2L4suX+s= X-Received: by 2002:a17:906:1a16:: with SMTP id i22mr5432371ejf.522.1615323174025; Tue, 09 Mar 2021 12:52:54 -0800 (PST) MIME-Version: 1.0 From: Duane Raymond <duane.raymond@fairsay.com> Date: Tue, 9 Mar 2021 21:52:36 +0100 Message-ID: <CAMs==2HNnd_6+4=iWZhK3d7oTvqk2W80wMdCJ+Y9eRFUKno1mA@mail.gmail.com> Subject: Test to identify shunting error To: test@campaigningforum.com Content-Type: multipart/alternative; boundary="000000000000fc503e05bd20bba9" Message-ID-Hash: 3THPVXHXTHRNBXZHN2TD5VBUWIMXOEY2 X-Message-ID-Hash: 3THPVXHXTHRNBXZHN2TD5VBUWIMXOEY2 X-MailFrom: duane.raymond@fairsay.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header

--000000000000fc503e05bd20bba9 Content-Type: text/plain; charset="UTF-8"

Test to identify shunting error

--000000000000fc503e05bd20bba9 Content-Type: text/html; charset="UTF-8"

<div dir="ltr">Test to identify shunting error<br></div>

--000000000000fc503e05bd20bba9--

<----- start object 2 -----> { '_parsemsg': False, 'envsender': 'noreply@campaigningforum.com', 'lang': 'en', 'listid': 'test.campaigningforum.com', 'original_size': 2436, 'received_time': datetime.datetime(2021, 3, 9, 20, 52, 54, 941548), 'rule_hits': [], 'rule_misses': [ 'dmarc-mitigation', 'no-senders', 'approved', 'emergency', 'loop', 'banned-address', 'member-moderation', 'nonmember-moderation', 'administrivia', 'implicit-dest', 'max-recipients', 'max-size', 'news-moderation', 'no-subject', 'digests', 'suspicious-header'], 'to_list': True, 'version': 3, 'whichq': 'pipeline'} [----- end pickle -----]

As I said there, Mailman is going through the roster of regular (non-digest) list members to find those with delivery enabled and it finds a member that has no associated address. Can you view the members of this list?

Yes - the list has three members (test list): me and two others. For my subscription I have gone through the membership options and explicitly set them. For the others their membership options are all unselected and thus I assume defaulting to a fallback value. (not sure if that matters)

Hi Mark,

I think I have it working thanks to some hints in your questions. here is the process I took in case others experience it the same or in case anyone wishes to comment!

When I use the mailman shell comment it seems to reflect just what you posted (I also tried test.campaigningforum.com and got the same result): (venv) mailman@mail:/opt/mailman/mm$ mailman shell -l test@campaigningforum.com Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the test@campaigningforum.com mailing list

...

...

Was there something beyond running this and getting the output I was supposed to enter? Was an error expected if one members doesn't have an associated address record?

The membership list is simply three emails persona@domaina.com persona_surname@domaina.com "Duane Raymond" <duane.raymond@fairsay.com>

There was one non-member address ("Duane Raymond" <duane.raymond@fairsay.com> - notice it is the same as the member address above) which I removed and the same member address above stayed listed.

I then sent another test email and still nothing was distributed or archived.

After searching around for a cause of the error message, I found this: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/V... which mentioning missing ros/values in the database tables... so then I took a different route.

I accessed the database tables directly to see if there was anything strange that related to your comment that "at least one of those members doesn't have an associated address record." What I found in the 'members' table were two rows which were not listed in the user listing of the mailing list GUI. So I deleted these and tried again.

This time the posting was archived and I did get an acknowledgement of the post, but despite having the option "Receive own postings" set to "yes", I didn't receive it. So I continued to investigate the database table and found that between the tables 'member' -> user and member -> address, the member table had the 'address_id' value but a null 'user_id' - so I fixed these and tried again. So far, there is still no email sent but archiving works fine and there is an acknowledgement email.

In the log files mail.info mail.log it seems it was sent as the message is: Mar 10 14:17:57 mail postfix/smtp[17747]: 844B74236D: to=<duane.raymond@fairsay.com>, relay=aspmx.l.google.com[2a00:1450:4013:c07::1a]:25, delay=0.6, delays=0.01/0/0.42/0.18, dsn=2.0.0, status=sent (250 2.0.0 OK 1615385877 t8si7488737ejj.661 - gsmtp) Mar 10 14:17:57 mail postfix/qmgr[4201]: 844B74236D: removed

So - in conclusion - the main problem now (after fixing the relationships in the tables which solved the archiving problem) seems to be that emails are not being received (and thus in a queue somewhere?) since the join email to the test list is sent and received immediately.

...sorry for the long post...but wanted to document what I've been trying and finding!

Duane

On 3/10/21 9:49 AM, Duane Raymond wrote:

Update: the other person on the test list has indicated they are receiving the emails now - and that gmail may be the cause of the issue by filtering out copies of the message I had sent. (my fairsay.com address is gmail hosted, but another is not and is simply forwarded there).

...so perhaps the problem is solved (will do a little more testing to verify!)

It's Gmail. Just check the mail.log and look for status=sent

-- Brian Carpenter Harmonylists.com Emwd.com

On 3/10/21 6:37 AM, Duane Raymond wrote:

Hi Mark,

I think I have it working thanks to some hints in your questions. here is the process I took in case others experience it the same or in case anyone wishes to comment!

When I use the mailman shell comment it seems to reflect just what you posted (I also tried test.campaigningforum.com and got the same result): (venv) mailman@mail:/opt/mailman/mm$ mailman shell -l test@campaigningforum.com Welcome to the GNU Mailman shell Use commit() to commit changes. Use abort() to discard changes since the last commit. Exit with ctrl+D does an implicit commit() but exit() does not. The variable 'm' is the test@campaigningforum.com mailing list

...

...

...

Was there something beyond running this and getting the output I was supposed to enter? Was an error expected if one members doesn't have an associated address record?

Did you enter this following the initial >>> prompt?

...

...

for mbr in m.members.members: ... if mbr.address is None: ... print('user: {}, user.addresses: {}'.format( ''' mbr.user, mbr.user.addresses)) ...

If so, it was expected to print the representation of the user object and that objects addresses list fo any member whose address attribute is None.

...

This time the posting was archived and I did get an acknowledgement of the post, but despite having the option "Receive own postings" set to "yes", I didn't receive it.

As you have determined, this is a googlemail issue. See <https://wiki.list.org/x/4030680>.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

On 3/10/21 2:32 PM, Duane Raymond wrote:

As for the potential bug in hyperkitty, I have been referencing the hyperkitty version of this thread as I was working the problem, and while I faintly remember seeing the loop in the email on my phone, when I was referencing the hyperkitty version of your email it was not there - even when expanding the quoted text....perhaps you see the same thing via hyperkitty? Essentially it removed part of the message that was intended to be seen.

That's a known issue with the HyperKitty that is installed at mailman3.org and python.org due to the patch for markdown rendering <https://gitlab.com/mailman/hyperkitty/-/merge_requests/160>. This patch is superceded by <https://gitlab.com/mailman/hyperkitty/-/merge_requests/324>, but I haven't gotten around to trying that yet.

If you look at the message view at <https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/ZZIAHLHDSJJ3AJEZ52VH3TOAE5XU6IDA/>, you'll see the missing part

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

I just installed the changes from https://gitlab.com/mailman/hyperkitty/-/merge_requests/324 on lists.mailman3.org and the thread view is now better.

Fantastic! Now it is readable via hyperkitty :-) (not 'perfect' but then I know parsing unstructured text perfectly when you don't know what it will contain is very difficult!)