mailman addmember/syncmember commands print undecoded UTF-8 [ADD] or [DEL] mail addresses in its messages like "[ADD] =?utf-8?q?John_=....."
My post here is merely about the information output when running mailman addmembers/syncmembers where I discovered strange looking lines.
I noticed that addmember and syncmember output looks like
[ADD] =?utf-8?q?John_=28Foo=29_Doe_=5Bcomment1=5D_=5Bcomment2=5D?= <john.foo.doe@example.com>
whereas the entries in the Web-GUI look fine. I triple-checked the textfile from which the names stem from, yes, the file is UTF8.
Is this behaviour of addmember/syncmember command output commonly known - or is this a small cosmetic bug?
PS. At some moment I had, probably from my test and legay data file, also these codings in about fifty entries of my users lists, but deleted that users with a small script, so that user list is now clean. My post here is merely about the information output when running mailman addmembers/syncmembers.
Wikinaut writes:
[ADD] =?utf-8?q?John_=28Foo=29_Doe_=5Bcomment1=5D_=5Bcomment2=5D?= <john.foo.doe@example.com>
That's called a MIME word, defined in RFC 2047. It's used when something in a messages can't be presented verbatim, in this case because the characters "()[]" have defined syntax in the context of an address, and therefore must be quoted in the display name ... but they weren't. It's a convenient way to ensure that the original display is reproduced exactly without Mailman adding quoting.
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan
On 8/5/25 02:50, Wikinaut wrote:
My post here is merely about the information output when running mailman addmembers/syncmembers where I discovered strange looking lines.
I noticed that addmember and syncmember output looks like
[ADD] =?utf-8?q?John_=28Foo=29_Doe_=5Bcomment1=5D_=5Bcomment2=5D?= <john.foo.doe@example.com>
whereas the entries in the Web-GUI look fine. I triple-checked the textfile from which the names stem from, yes, the file is UTF8.
What was the exact line in the textfile for this address?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I have such entries again in the postorius list "=?utf-8?q?Alexandro....", even after having deleted the user twice (there were no entries), the text file has nothing special, uchardet says "UTF8".
On 8/5/25 12:13, Wikinaut wrote:
I have such entries again in the postorius list "=?utf-8?q?Alexandro....", even after having deleted the user twice (there were no entries), the text file has nothing special, uchardet says "UTF8".
Presumably these are being picked up from an email header that has the display name properly RFC 2047 encoded but then not decoded for the database address or user record. This could be a bug, but without knowing the exact scenario that results in these encoded display names, it's difficult to know where the bug might be.
In what postorius list (members or non-members) are you seeing these? Are there log messages about them in the subscribe log (by default written to mailman.log)?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Not a single mail was sent - I am migrating (and newly setting up from textfiles) the mailman3 suite, installed fresh from the scratch in (venv) in /opt/mailman.
I noticed that after having used a mix of addmember and also syncmember, one (or yesterday about 100 entries) of 500 had the MIME-coding. Then I used a script to remove users with the "=?UTF...." begin. In postorius users the user was gone and also in the lists where this user was in.
def delete_users(): url = f"{API_URL}/users" r = requests.get(f'{url}', auth = AUTH) r.raise_for_status() users = r.json()['entries']
for user in users:
dn = user.get('display_name')
if type(dn) == type("string") and dn.startswith("="):
print (dn)
self_link = user.get('self_link')
requests.delete(self_link, auth=AUTH)def main(): delete_users()
After a new add/syncmember, that spooky entry was up again!
Then I used mailman delmembers --fromall --no-goodbye-msg --no-admin-notify -m <this@address>: Then after a new add/syncmember, that entry was gone and the correct entry was shown.
That means, that somewhere some remains were stored somewhere. I also have a another issue, where I cannot get rid of a test display_name like "1111 [äöo]" <myemailaddress> for my own user entry (this issue has not yet been filed, as I do not want to flood you here; the "=?utf.." issue seems to be a more apparent problem).
At the moment, none of the ~500 users have "utf" searchable, which is some good news.
Perhaps to mention: I noticed the utf-problems when I used Ctrl-C during heavy script actions while add/syncing the lists which I needed to stop. while testing.
Perhaps this caused an inconsistent database state??? and that present issue was a consequence of my harsh break?
On 8/5/25 13:12, Wikinaut wrote:
After a new add/syncmember, that spooky entry was up again!
And were any of the spooky entry email addresses in the input to that command? If so what was the exact display name for the line containing that address. If that looked normal,
Then I used mailman delmembers --fromall --no-goodbye-msg --no-admin-notify -m <this@address>: Then after a new add/syncmember, that entry was gone and the correct entry was shown.
That means, that somewhere some remains were stored somewhere.
Possibly in an Address record. Display names are in User Records and Address records. The script you ran to delete users should have removed User records, but an Address record for the address with the odd display name may have remained.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
The script you ran to delete users should have removed. User records, but an Address record for the address with the odd display name may have remained.
This sounds very reasonable, may be the reason. I will modify deleteUsers to also remove addresses (may be that your regular delmembers removes members&addresses&users under the hood?) I should have better used delmember but this was not possible for delmembers with a string search in the display_name.
Dear Mark,
the problem has not gone. Now, in production during regular list upgrades, I got one member with 2(two) entries, one regularly encoded, the other double-escaped (can we call that so?).
If you send me your private address, I can send you a screenshot.
On 8/6/25 04:44, Wikinaut wrote:
Dear Mark,
the problem has not gone. Now, in production during regular list upgrades, I got one member with 2(two) entries, one regularly encoded, the other double-escaped (can we call that so?).
I'm interested in exactly what is done to produce these entries. Are you
running mailman syncmembers or mailman addmembers? If so what is the
input file that produces this. You can send it to me off list.
If you send me your private address, I can send you a screenshot.
You can email me off list at the address in this and all my other posts.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
Mark Sapiro -
Stephen J. Turnbull -
Wikinaut