Search results for query "sapiro"
- 5707 messages
[MM3-users] Re: Fwd: [Django] ERROR (EXTERNAL IP): Internal Server Error: /mailman3/hyperkitty/api/mailman/archive
by David Partain
On Thu, 2021-01-28 at 12:03 -0800, Mark Sapiro wrote:
On 1/28/21 5:33 AM, David Partain via Mailman-users wrote:
I wrote:
I have the same issue, and have changed the configuration in /etc/mailman3/mailman-web.py to:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': 'mailman3web',
'USER': 'mailman3web',
'PASSWORD': 'xxxxxxxxxxxxxxxxxx',
'HOST': 'localhost',
'PORT': '',
'OPTIONS': {
'init_command': "SET sql_mode='STRICT_TRANS_TABLES'",
'charset': 'utf8mb4',
},
}
}
and restarted stuff, but I cannot tell that it has changed anything.
You responded:
That should do it. I think the original message should have been saved
as a queue entry in Mailman's var/archives/hyperkitty/spool/ directory.
That directory has all of the messages sent to that list:
# ls /var/lib/mailman3/archives/hyperkitty/spool/
1611827925.8030825+925c996b9b6020ee18e9a689ce23384855159f77.pck 1611827926.1923075+e728f3c29c1fe6eeeca641314cdba7007cb6e132.pck
1611827925.8917305+d16b7004bdbbef98ce40fb8ee70e492715d1607d.pck 1611827926.2623458+1aac13b50d8e7f32eb14d13e6559656aa8f3bd10.pck
1611827925.9594557+7ad2a910b5ee9142c88f0bf26c60c6df960d4262.pck 1611827926.3488555+04a540651c5f829751518433e1c54168ab63a38b.pck
1611827926.0404096+a5e745f51a9f7d541aeb770a89604be17019f3c5.pck 1611827926.4206145+de861eec6404d0c8f9f8bdb180e29bff2de7c52b.pck
1611827926.1167526+0aa63639a4ec102bf63522827637e75e6a2b092a.pck
This is what's expected since the underlying issue isn't yet fixed.
Yes, I'm still seeing the errors. When I send mail to a list with a cute emoji, I see the following in /var/log/mailman3/web/mailman-web.log:
ERROR 2021-01-28 10:58:45,708 16923 django.request Internal Server Error: /mailman3/hyperkitty/api/mailman/archive
(bunch of python errors omitted)
DataError: (1366, u"Incorrect string value: '\\xF0\\x9F\\x98\\x8A\\x0A\\x0A...' for column 'content' at row 1")
OK. The issue is with your database it doesn't like '\xF0\x9F\x98\x8A'
which is a 4-byte utf-8 encoding for a smiley face emoji.
The issue is with MySQL and MariaDB which do not accept 4-byte utf-8
encodings by default. The settings you show above should be sufficient
to allow 4-byte utf-8 encodings.
I'm not that familiar with how the Debian/Ubuntu package sets things up.
Are you sure that /etc/mailman3/mailman-web.py is the settings file used
by Django, and if so does it have at the end something like
```
try:
from settings_local import *
except ImportError:
pass
```
and if so is there something there that overrides the DATABASES setting?
If all that is OK, you may need to do something like
```
ALTER DATABASE mailman3web CHARACTER SET utf8mb4;
ALTER TABLE hyperkitty_email CHARACTER SET utf8mb4;
```
on the database itself.
Hi Mark (and others),
I'm now looking at this again after having to focus on other stuff. I'm befuddled, and I don't really know what to do now.
I did the above.
If I send mail to my list with those bleeping emojis, I see the same errors. In /var/log/mailman3/web/mailman-web.log I see
DataError: (1366, u"Incorrect string value: '\\xF0\\x9F\\x95\\xB5\\xF0\\x9F...
new mail to root, /var/lib/mailman3/archives/hyperkitty/spool/ has a new file, etc.
The mail gets delivered, of course.
If I then look at the database itself, I see:
mysql> use mailman3web;
mysql> SELECT @@character_set_database, @@collation_database;
+--------------------------+----------------------+
| @@character_set_database | @@collation_database |
+--------------------------+----------------------+
| utf8mb4 | utf8mb4_general_ci |
+--------------------------+----------------------+
1 row in set (0.00 sec)
mysql> SHOW TABLE STATUS where name like 'hyperkitty_email';
+------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length | Data_free | Auto_increment | Create_time | Update_time | Check_time | Collation | Checksum | Create_options | Comment |
+------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
| hyperkitty_email | InnoDB | 10 | Dynamic | 133 | 11949 | 1589248 | 0 | 196608 | 4194304 | 150 | 2021-01-29 17:23:13 | NULL | NULL | utf8mb4_general_ci | NULL | | |
+------------------+--------+---------+------------+------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+-------------+------------+--------------------+----------+----------------+---------+
1 row in set (0.00 sec)
So, I go look elsewhere...
In /etc/mailman3/mailman.cfg I see:
class: mailman.database.mysql.MySQLDatabase
url: mysql+pymysql://mailman3:NopeNotPastingThat@localhost/mailman3?charset=utf8&use_unicode=1
Should I change the latter to be something like:
url: mysql+pymysql://mailman3:NopeNotPastingThat@localhost/mailman3?charset=utf8mb4&use_unicode=1
?
Cheers,
David
4 years, 4 months
[MM3-users] Re: Join a Mailman 3 list?
by Thomas Ward
On 2025-06-02 20:19, Cathryn McGuire wrote:
> We are Administrators of a small neighbourhood group with more than 20 years of archives.
You'll have to talk to EMWD to see if your archives will be kept with
the migration. They have a completely custom web interface for archival.
---
I'm going to re-state what Mark said here. To quote Mark Sapiro:
> As far as your EMWD migration from cPanel Mailman 2.1 to Mailman 3 is
conerned, I think you'll find little if any change as far as email
posting to the list and email from the list is concerned, but EMWD has
their own proprietary web UI for list management and archiving, so the
web UI at EMWD is quite different from that for this list.
THEREFORE...
> We love the simple functionality of Mailman 2.
> One post goes to the entire membership by email. Any reply arrives by email; either immediately or as part of the 9:00 am Digest.
That part isn't going to change.
> No websites or browsers or sign-ins.
Well, you manage your list SOMEHOW, probably with EMWD's web panel and
UI panel.
> No threads or topics.
There's no "threads" like forums have, but discussions by topic at least
in mailman 2 and mailman 3 have still existed and probably still will
exist.
> Simply emails back and forth.
>
> Does Mailman 3 have the ability to function this way?
(Read my last statements)
> Might someone invite us to temporarily join their group so we can see how Mailman 3 operates from the point of view of members?\
EMWD uses their **own custom UI** for management and archiving. This
means that we can't give you any real guidance on EMWD and such for how
it relates to user management, etc. because the standard Mailman 3 UI
(Postorious) for that won't be what you use with EMWD. So we can't
really assist you to see how things will be from the view of "members of
the list" in that front.
For the vast majority of users and use cases though, if all they care
about is the email messages being sent to the list and received from the
list, nothing will change for users. It's the *management* components
and EMWD's custom UIs you'll have to work with though, not the standard
Postorious Mailman3 UI.
At my dayjob, we use mailing lists here on Mailman3, albeit with the
standard Postorious UI, but we have a separate management UI that we use
for managers to check who is a member, remove members, etc. that uses
the Mailman API and not Postorious, but the core management group
(myself and IT staff) use Postorious for our management tasks and to see
if Mailman is working and such. Additionally, we have 53 lists and well
over a thousand members on some of othe lists and it works well. The
vast majority of the members only care that the email functionality
works, and it does for them. (None of them use digest format mail
notifications, but that's because the lists tend to be more 'realtime'
for those members.) (And before you ask, those lists are not open)
If you have *more specific* questions beyond "do the same basics of
Mailman2 like digests, email handling, etc. still behave more or less
the same way" then you should ask those questions *specifically*.
If your specific questions revolve around the management interface, you
have to talk to EMWD. Same with message archives.
Thomas
4 weeks
[MM3-users] Re: Templates
by David Krantz
Hi again,
I had the problem with getting two footers, i.e. two expansions of the
template. After some digging in the source code I found out that the
decorator made one expansion (without personalization) and that the
delivery handling made another (with personalization). So I hacked
pipelines/builtin.py with a new pipeline that excluded the decorate
step, recompiled and changed to that pipeline for my personalized
list. That worked. Nothing like a bit of shotgun debugging in
production.
That solves my immediate problem. Is there a known and recommended way
of configuring personalized delivery so that you only get the
personalized footer? I have ideas, but I probably should share them
with the developer list instead. :-)
Of course this very nice functionality is a bit buried. I activated it
using the interactive python shell, seems less risky than changing the
database directly.
Cheers
// David
On Sat, Sep 23, 2017 at 10:39 PM, David Krantz <gazorg(a)gmail.com> wrote:
> On Sat, Sep 23, 2017 at 6:48 PM, Mark Sapiro <mark(a)msapiro.net> wrote:
>>
>> Have you tried restarting/reloading whatever dbm (MySQL, PostgreSQL, ?)
>> you are using?
>
> No, that would have been strange to me; the table in the database was
> up to date. But now I tried that, I got no effect this time but I
> probably could set up a better test.
>
> I also had another problem [solved]: I get the error message "Multiple
> rows were found for one_or_none" when I try TemplateManager.get(name,
> context) for the list. There is only one matching row in the templates
> table so something is fishy with the ORM. This occured when I changed
> to the previous uri value. As the database does not have any
> duplicates in the templates table I suspect either a broken join or
> that the cache retains the old value.
>
> The traceback looks like this in the mailman shell:
>>>> tm.get('list:member:regular:footer', 'test.example.com')
> Traceback (most recent call last):
> File "<console>", line 1, in <module>
> File ".../venv-3.5/lib/python3.5/site-packages/mailman-3.2.0a1-py3.5.egg/mailman/database/transaction.py",
> line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File ".../venv-3.5/lib/python3.5/site-packages/mailman-3.2.0a1-py3.5.egg/mailman/model/template.py",
> line 98, in get
> contents = cache_mgr.get(actual_uri)
> File ".../venv-3.5/lib/python3.5/site-packages/mailman-3.2.0a1-py3.5.egg/mailman/database/transaction.py",
> line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File ".../venv-3.5/lib/python3.5/site-packages/mailman-3.2.0a1-py3.5.egg/mailman/model/cache.py",
> line 125, in get
> CacheEntry.key == key).one_or_none()
> File ".../venv-3.5/lib/python3.5/site-packages/SQLAlchemy-1.2.0b2-py3.5-linux-x86_64.egg/sqlalchemy/orm/query.py",
> line 2814, in one_or_none
> "Multiple rows were found for one_or_none()")
> sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for
> one_or_none()
>
> Workaround: Delete cached files in .../var/cache and run delete from
> file_cache in postgresql. Then that works again. The culprit was that
> the file got cached twice so there is a bug in the cache function. I
> used the ITemplateManager-util in the shell to cause the problem.
>
>>> 2. the $user_delivered_to and $user_email do not expand for some
>>> reason. Known problem?
>>
>>
>> These require that the list delivery be personalized. See
>> <http://mailman.readthedocs.io/en/latest/src/mailman/runners/docs/outgoing.h…>.
>>
>
> That explains things. Thank you!
>
> Now I got two footers, first one without personalisation and one
> personalised. Interesting.
>
> best regards
> // David
7 years, 9 months
[MM3-users] Re: psycopg2 error?
by Odhiambo Washington
On Fri, Oct 27, 2023 at 8:21 PM Mark Sapiro <mark(a)msapiro.net> wrote:
> On 10/27/23 09:16, Odhiambo Washington wrote:
> > (venv) mailman@debian12:~$ pip cache remove psycopg2-binary
> > WARNING: No matching packages for pattern "psycopg2-binary"
> > Files removed: 0
>
> This was intended to prevent the following 'Using cached'.
>
>
> > (venv) mailman@debian12:~$ pip install psycopg2-binary
> > Collecting psycopg2-binary
> > Using cached
> >
> psycopg2_binary-2.9.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> > (3.0 MB)
>
>
> However, you now have a different issue.
>
>
> > raise DistributionNotFound(req, requirers)
> > pkg_resources.DistributionNotFound: The 'flufl.lock>=5.1' distribution
> was
> > not found and is required by mailman
>
> See https://gitlab.com/mailman/mailman/-/issues/1085
Thank you. That solves the problem for sure! But this was detected 4 months
ago. When is the permanent solution coming?
(venv) mailman@debian12:~$ vi
venv/lib/python3.11/site-packages/mailman-3.3.9.egg-info/requires.txt #
flufl.xxx to flufl-xxx
(venv) mailman@debian12:~$ mailman info
GNU Mailman 3.3.9 (Tom Sawyer)
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
config file: /etc/mailman3/mailman.cfg
db url: postgresql://mailman:XXXXXXX@localhost/mailman
devmode: DISABLED
REST root url: http://localhost:8001/3.1/
REST credentials: restadmin:restpass
(venv) mailman@debian12:~$ exit
logout
root@debian12:/home/wash# systemctl start mailman3
root@debian12:/home/wash# systemctl status mailman3
● mailman3.service - GNU Mailing List Manager
Loaded: loaded (/etc/systemd/system/mailman3.service; enabled; preset:
enabled)
Active: active (running) since Sat 2023-10-28 01:22:56 EAT; 7s ago
Process: 15590 ExecStart=/opt/mailman/venv/bin/mailman start
(code=exited, status=0/SUCCESS)
Main PID: 15592 (python3)
Tasks: 17 (limit: 4547)
Memory: 980.8M
CPU: 27.774s
CGroup: /system.slice/mailman3.service
├─15592 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/master -C /etc/mailman3/mailman.cfg
├─15598 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg
--runner=archive:0:1
├─15599 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg
--runner=bounces:0:1
├─15600 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg
--runner=command:0:1
├─15601 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=in:0:1
├─15602 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=lmtp:0:1
├─15603 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=nntp:0:1
├─15604 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=out:0:1
├─15605 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg
--runner=pipeline:0:1
├─15606 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1
├─15607 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=retry:0:1
├─15608 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=task:0:1
├─15609 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg
--runner=virgin:0:1
├─15610 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg
--runner=digest:0:1
├─15648 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1
└─15649 /opt/mailman/venv/bin/python3
/opt/mailman/venv/bin/runner -C /etc/mailman3/mailman.cfg --runner=rest:0:1
Oct 28 01:22:54 debian12.wash.lan systemd[1]: Starting mailman3.service -
GNU Mailing List Manager...
Oct 28 01:22:55 debian12.wash.lan mailman[15590]: Starting Mailman's master
runner
Oct 28 01:22:55 debian12.wash.lan mailman[15590]: Generating MTA alias maps
Oct 28 01:22:56 debian12.wash.lan systemd[1]: mailman3.service: Can't open
PID file /opt/mailman/mm/var/master.pid (yet?) after start: No such file or
directory
Oct 28 01:22:56 debian12.wash.lan systemd[1]: Started mailman3.service -
GNU Mailing List Manager.
root@debian12:/home/wash#
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
[How to ask smart questions:
http://www.catb.org/~esr/faqs/smart-questions.html]
1 year, 8 months
[MM3-users] Re: Different Mail Domain Cause Key Error
by Brian Carpenter
On 4/21/20 4:04 PM, Mark Sapiro wrote:
> This is not the error. The KeyError ... 'file' exception is a Django
> logging exception. It is not the underlying error. There should be more
> in the log indicating what the real problem is.
Ok. The first thing however is did I miss a step?
Here is the complete error that is sent to me via email as superuser:
Internal Server Error:/mailman3/lists/
KeyError at/mailman3/lists/
'file'
Traceback:
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
115. response = self.process_exception_by_middleware(e, request)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
113. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/postorius/views/list.py" in list_index
755. paginator_class=MailmanPaginator)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django_mailman3/lib/paginator.py" in paginate
71. objects = paginator.page(page_num)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django_mailman3/lib/paginator.py" in page
46. number = self.validate_number(number)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/paginator.py" in validate_number
48. if number > self.num_pages:
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/utils/functional.py" in __get__
80. res = instance.__dict__[self.name] = self.func(instance)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/paginator.py" in num_pages
97. if self.count == 0 and not self.allow_empty_first_page:
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/utils/functional.py" in __get__
80. res = instance.__dict__[self.name] = self.func(instance)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django_mailman3/lib/paginator.py" in count
56. return self.function(count=0, page=1).total_size
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/postorius/views/list.py" in _get_list_page
751. advertised=advertised, mail_host=mail_host, count=count, page=page)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/client.py" in get_list_page
181. return Page(self._connection, url, MailingList, count, page)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/page.py" in __init__
37. self._create_page()
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/page.py" in _create_page
62. response, content = self._connection.call(self._build_url())
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/mailmanclient/restbase/connection.py" in call
112. error_msg, response, None)
*During handling of the above exception (HTTP Error 404: {"title": "404
Not Found"}), another exception occurred:*
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/handlers/exception.py" in inner
34. response = get_response(request)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/postorius/middleware.py" in __call__
42. return self.get_response(request)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/handlers/exception.py" in inner
36. response = response_for_exception(request, exc)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/core/handlers/exception.py" in response_for_exception
95. exc_info=sys.exc_info(),
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/utils/log.py" in log_response
228. exc_info=exc_info,
File "/usr/lib/python3.7/logging/__init__.py" in error
1412. self._log(ERROR, msg, args, **kwargs)
File "/usr/lib/python3.7/logging/__init__.py" in _log
1519. self.handle(record)
File "/usr/lib/python3.7/logging/__init__.py" in handle
1529. self.callHandlers(record)
File "/usr/lib/python3.7/logging/__init__.py" in callHandlers
1591. hdlr.handle(record)
File "/usr/lib/python3.7/logging/__init__.py" in handle
905. self.emit(record)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/utils/log.py" in emit
119. reporter = ExceptionReporter(request, is_email=True, *exc_info)
File "/opt/mailman/mm/venv/lib/python3.7/site-packages/django/views/debug.py" in __init__
254. self.template_info = getattr(self.exc_value, 'template_debug', None)
File "/usr/lib/python3.7/tempfile.py" in __getattr__
614. file = self.__dict__['file']
Exception Type: KeyError at/mailman3/lists/
Exception Value: 'file'
Request information:
USER: AnonymousUser
--
Please let me know if you need further assistance.
Thank you for your business. We appreciate our clients.
Brian Carpenter
EMWD.com
--
EMWD's Knowledgebase:
https://clientarea.emwd.com/index.php/knowledgebase
EMWD's Community Forums
http://discourse.emwd.com/
5 years, 2 months
[MM3-users] [SOLVED] Re: Re: Newbie question 2: Rewriting issue for bounced emails
by William Oliver
On Mon, 2021-12-27 at 14:36 -0800, Mark Sapiro wrote:
>
> It appears that the MX for billoblog.com is not accepting mail from
> your
> server. Can you send non-mailman mail from this server to this address?
> If not, you need to contact billoblog.com to fin wht they are blocking
> your mail.
>
> > I tried changing the smtp port in mailman.cfg to 587, but that didn't
> > change anything.
>
> because the issue is not in delivery from Mailman to Postfix. It is in
> delivery from Postfix to billoblog.com.
First, thanks so much for talking me through this. I could never have
done it on my own. If you ever decide to go to the Smoky Mountain
National Park or Knoxville, TN, send me an email, and I'll buy you a
drink (I live in the hills between Knoxville and Gatlinburg, TN).
In any case, I did some wandering around the intertubes and it appears
this is an uncommon but well-known problem when using dovecot and
postfix. It has to do with the setting of
smtpd_recipient_restrictions
in master.cf.
It's not clear *which* "reject" command is triggered, but if I set it
to just:
-o smtpd_recipient_restrictions=permit_sasl_authenticated
then it can get through. Er, to the next error, anyway. Sigh.
I have *no* idea why it only shows up with mailman, but not with
regular mail. I got some learnin' to do about that line.
I don't know what the default reject criteria are. I have this
horrible suspicion that I've opened myself up to zillions of evil
emails, but I'll investigate that later.
So, this took care of the Access Denied issue. That opened up the next
error, which was "relay access denied". Again, regular mail worked
just fine.
Here's the syslog when I sent email to testlist from
fp145(a)libertyfp.org:
Here's the rejection to billo(a)billoblog.com:
Dec 27 18:31:52 libertyfp postfix/submission/smtpd[471207]: connect
from mail.libertyfp.org[2.56.57.28]
Dec 27 18:31:52 libertyfp postfix/submission/smtpd[471207]: NOQUEUE:
reject: RCPT from mail.libertyfp.org[2.56.57.28]: 454 4.7.1
<billo(a)billoblog.com>: Relay access denied;
from=<testlist-bounces+billo=billoblog.com(a)libertyfp.org>
to=<billo(a)billoblog.com> proto=ESMTP helo=<mail.libertyfp.org>
Dec 27 18:31:52 libertyfp postfix/submission/smtpd[471207]: disconnect
from mail.libertyfp.org[2.56.57.28] ehlo=1 mail=1 rcpt=0/1 rset=1
quit=1 commands=4/5
But it accepts the email to fp145(a)libertyfp.org
Dec 27 18:31:53 libertyfp postfix/submission/smtpd[471207]: connect
from mail.libertyfp.org[2.56.57.28]
Dec 27 18:31:53 libertyfp postfix/submission/smtpd[471207]: 4FCDD420F5:
client=mail.libertyfp.org[2.56.57.28]
Dec 27 18:31:53 libertyfp postfix/cleanup[471223]: 4FCDD420F5: message-
id=<8d3bf47c5dcd86983e8c01f8baece4e3ac43a1a1.camel(a)libertyfp.org>
Dec 27 18:31:53 libertyfp postfix/cleanup[471223]: 4FCDD420F5: warning:
header Subject: [Testlist] bbbbb from mail.libertyfp.org[2.56.57.28];
from=<testlist-bounces+fp145=libertyfp.org(a)libertyfp.org>
to=<fp145(a)libertyfp.org> proto=ESMTP helo=<mail.libertyfp.org>
Dec 27 18:31:53 libertyfp postfix/qmgr[468573]: 4FCDD420F5:
from=<testlist-bounces+fp145=libertyfp.org(a)libertyfp.org>, size=1870,
nrcpt=1 (queue active)
Dec 27 18:31:53 libertyfp postfix/submission/smtpd[471207]: disconnect
from mail.libertyfp.org[2.56.57.28] ehlo=1 mail=1 rcpt=1 data=1 quit=1
commands=5
Direct mail from fp145(a)libertyfp.org to billo(a)billoblog.com worked fine
also.
Again, a quick DuckDuckGo search reveals that this has come up a couple
of times. It required that I modify /etc/postfix/main.cf to add the
non-localhost ip4 address of the box to the mynetworks line:
mynetworks = 127.0.0.0/8 10.0.0.0/24
to
mynetworks = 127.0.0.0/8 10.0.0.0/24 2.56.57.28
I assume that means that postfix uses localhost for regular mail, but
mailman3 uses the 2.56.57.28 address.
In any case, it now works fine. I am happily receiving the six million
test messages that were queued awaiting delivery to
billo(a)billoblog.com.
Once again, thanks. I owe you.
billo
3 years, 6 months
[MM3-users] Re: Splitting hosts: mta&core vs postorius
by Andrew Hodgson
Stephen J. Turnbull wrote:
>Ruth Ivimey-Cook writes:
> > On 21/06/2020 04:52, Mark Sapiro wrote:
> > > Change that to greyarea-post
> > Problem is, if I do that (which I did in the beginning), the core rest > service listens on localhost / 127.0.0.1
>I don't understand. I'm not in a position to test with Mailman itself right now, but the standalone gunicorn[1] does the right thing (takes an IP address and binds to the corresponding interface, or takes a domain name, resolves it to >the IP address, and binds that to the corresponding interface). It does not substitute localhost.
Sorry not to reply to ruth on this before, my main experience of this issue was running in a Kubernetes cluster using Docker, using https://github.com/maxking/docker-mailman as the basis for the images. What follows is my own research on this issue and it may be completely wrong.
Taking the default Mailman Core setting:
[webservice] hostname: localhost
This makes Mailman-Core listen on the loopback address (via name lookup) on the configured port for incoming REST requests. When it responds to these requests, it sends out in the response a self_link address which matches the [webservice] hostname Parameter. Connecting clients (such as Postorius) need to connect to Mailman-Core on the given hostname (eg http://localhost in the default case where Mailman Core and Web components are installed to the same machine.
However if the [webservice] hostname: parameter is set to something like 0.0.0.0, then although you can direct Postorius (via DNS lookup or IP address) at the Mailman Core instance, it fails to work properly because the resulting response sent back from Mailman Core will include self_links pointing at 0.0.0.0 which will not be usable as a URL by the REST client.
A fix for this issue is to set the [webservice] hostname parameter to a FQDN or IP address which can be resolved by the machine running the Web components, and as long as the base URL that Postorius tries to use for the Mailman Core access is the same as in the [webservice] hostname parameter, you are good to go.
It appears that when specifying a FQDN in the [webservice] hostname parameter, that the system will look up the IP address for that host using the systems name resolution function. This has the side affect of potentially listening on the wrong interface (localhost) if the system hosts file is set such that the machine's FQDN points to 127.0.0.1. In my case my containers were behind CoreDNS and this wasn't an issue for me, but had I tried this for example on a Debian system I would have come against the same issue that Ruth has as it will automatically set the FQDN of the system to resolve to 127.0.0.1 by way of the Hosts file. Adding the network IP address of the system with the FQDN in the hosts file should fix this issue.
> > IMO, the hostname parameter needs to be split up into a "what address > should the rest of the world use to address me" and separately "what > address(es) should I listen on". Multiple listen addresses enable > multihomed >hosts to get the right connectivity, too.
>I'm not clear on what you're saying. An address is a rendezvous point; if you're not listening on the address that others are using, you won't meet. What good does it do to split them up?
I agree with this, we need a setting for which interface for Gunicorn to listen on, and what Mailman Core sends out in the self_link attribute when responding to REST requests.
Thanks.
Andrew.
5 years
[MM3-users] Re: Help with some 'withlist' configuration changes
by David Partain
Hi!
Thank you, Mark, for giving me something to go on! I'll dig further.
Cheers,
David
On Wed, 2021-06-23 at 06:59 -0700, Mark Sapiro wrote:
On 6/23/21 6:30 AM, David Partain via Mailman-users wrote:
Hi,
Firstly, a repeat that I have a relatively broken (and soon to be fixed, I hope) installation of mailman3 based on Ubuntu 18.04 distrib packages (3.1.1-9) that stumbles along by me dealing with a bunch of things by hand that I hope'll go away once I'm up to the current version.
In this set-up, I get a fair number of items shunted. Since I have this wonky version where stuff isn't always exposed in the web interface, I have to fix these things by hand. An example is that something gets shunted because of too many recipients, where I do:
sudo -u list mailman withlist somemailmanlist
m.max_num_recipients = 0
commit()
sudo -u list mailman unshunt
I've managed to find a number of these, such as m.max_message_size, m.require_explicit_destination, and m.administrivia.
There is some other issue here as these things should only cause a
message to be held, not shunted. Some exception is being thrown in the
process of holding the messgage, and thaty is the real issue. What are
the tracebacks in Mailman's mailman.log for these shunts?
My questions are undoubtedly foolish, but here goes:
1. Is there any way to print the entire 'm' structure? If so, I'd perhaps be able to figure out the answers to my other questions.
dir(m)
or maybe you want
>>> for x in dir(m):
... if not x.startswith('_'):
... print(f'{x}: {m.__getattribute__(x)}')
...
2. Is there a way for me to correct the following shunt-ing issues with 'withlist'? I've been unable to figure out how...
a. 'moderation_reasons': [ 'Message has no subject'] / 'rule_hits': ['no-subject']
b. Some things get caught as 'rule_hits': ['nonmember-moderation']. However, if I look in the (web-based) config, it shows that non-members should be accepted but the messages are shunted nonetheless. I don't know what to look for in 'm' or how I might clear these messages.
As I note above, these things are reasons to hold the message. The
shunting occurs because something is going wrong in the holding process.
Find the errors and tracebacks in Mailman's mailman.log to see what the
real issue might be.
3. I have no idea what these are telling me to do about digest messages. Suggestions? This is the entire info I get from 'mailman qfile':
------------------------------------------------------snip snip----------------------------------------------
[----- start pickle -----]
<----- start object 1 ----->
<----- start object 2 ----->
{ '_parsemsg': False,
'digest_number': 457,
'digest_path': '/var/lib/mailman3/lists/somelist/digest.31.457.mmdf',
'lang': 'en',
'listid': 'somelistid',
'version': 3,
'volume': 31,
'whichq': 'digest'}
[----- end pickle -----]
------------------------------------------------------snip snip----------------------------------------------
That is telling you there was in issue in sending the digest, the
mailbox for which is
/var/lib/mailman3/lists/somelist/digest.31.457.mmdf. Unshunting that
will retry sending that digest.
4. I have some that have 'rule_hits': [] (it's empty) but are shunted nonetheless. I can't see anything weird other than one of the messages has a utf-8 subject. Any ideas?
5. Some messages that were sent to -request addresses (e.g., subscribe messages, confirm, approve messages) are being shunted. What should I do with these?
You need to find the errors in the log and figure out what the
underlying issue is. I'm guessing it's a permissions issue, possibly
because of having at one time rum `mailman start` as root causing files
to be created and owned by root and not be readable/writable by the
Mailman user.
4 years
[MM3-users] Re: mailman3 does not send to the members of the lists
by Stephen J. Turnbull
Mark Sapiro writes:
> I say you have to understand what you want Postfix to do and configure
> it appropriately.
I would recommend going to a Postfix list as Mark suggested. Postfix
configuration is complex, and they're more likely to know the best way
to do exactly what you say you want to do. I myself can't help with
Postfix details, I'm an Exim user. But my list of "need to know"
stuff is generic, just the answers would be very different with Exim
or Sendmail or qmail!
> I would need much more information in order to know how your Postfix
> should be configured,
In particular, whoever answers your questions will need to know:
1. Why are you using virtual domains for email? This doesn't need to
have great detail. Mostly:
- are you supporting multiple organizations with their own domains, OR
- is this for some purpose of your own, OR
- did you just copy the virtual domain lines from some example
configuration?
In the last case, you might try not even mentioning that you're
using virtual domains. (That's only if you treat this as a "new
install". If you're going to send them your current configs,
they'll see your virtual domain stuff and want to know "why?" You
may as well tell them.) They may tell you that you can do
everything you want without them! Or they may tell you that
virtual domains are the best way to do that, and you can be
confident about it.
In the last case you could also try just deleting and seeing if
your system works, as you suggested to Mark earlier. But that
could also break everything related to mail on that host.
2. Who is receiving mail via Dovecot, and via what domain(s)? The
"who" is generic (eg, "users and web applications using IMAP
clients"), but the domains should be specific.
If not inconvenient, it is easiest if you give real names for the
domains. If there are privacy, security, or intellectual property
concerns with the domain names, you'll need to find out the
conventions used by the Postfix community for dealing with that.
If you do redact your domain names in your explanation in the
initial contact, make sure you explain exactly how you did that,
or you will confuse everybody and have an unpleasant experience.
3. What domain does Mailman use to receive mail?
Is Mailman the only thing that receives mail on that domain?
What else, if anything, needs to received mail addressed to that
domain?
4. What domains other than those handled by Mailman and Dovecot, if
any, are processed by your Postfix MTA?
5. What other things, if any, besides Mailman and Dovecot directly
receive mail processed by your Postfix MTA?
Anything downstream from Mailman and Dovecot doesn't need to be
mentioned. If there are still problems with downstream recipients
once Mailman and Dovecot are receiving mail correctly, direct
those issues to us and to Dovecot, as appropriate.
You should check the channel FAQ to see whether they want your
configuration in your first contact, or not. Even if they do, you
might be better off to treat this as a "new install" rather than "it's
broken, help me fix it".
In contacting the Postfix channel, explain "I want to ..." based on
the answers to those questions. Then your question is "how do I
configure mail so that Dovecot domains and users get mail, Mailman
domains, lists, and administrators get mail, and any other mail
handled by Postfix goes to the right place (as defined by questions 4
and 5)?"
Steve
4 years, 11 months
[MM3-users] Re: Problems with New MM3 Installation
by Abhilash Raj
On Tue, May 11, 2021, at 8:43 PM, Mark Sapiro wrote:
> On 5/11/21 7:46 PM, Jeff wrote:
> >
> > I really appreciate your assistance. New messages are now being archived as expected, but I still have a few small remaining issues to work out with my MM3 installation:
> >
> > 1) some emails that arrived before I got the archive working are still sitting in:
> >
> > /usr/local/mailman3/mm/var/archives/prototype/[listname]@[domain]/new
> >
> > Strangely this includes at least one email that was archived. Should this directly be cleared as messages are archived? Is there a way to force it to be cleared?
>
> /usr/local/mailman3/mm/var/archives/prototype/[listname]@[domain] is a
> collection of messages in maildir format. This is how the prototype
> archiver stores messages. This has nothing to do with HyperKitty. It is
> a separate archiver. If you don't want messages to be stored there, you
> can disable the prototype archiver for the list by unchecking prototype
> in Postorius -> Settings -> Archiving -> Active archivers or disable it
> entirely for the installation by setting
> ```
> [archiver.prototype]
> enable: no
> ```
> in mailman.cfg
>
> > 2) Some messages that arrived while I was having other problems with the setup have been stored in:
> >
> > /usr/local/mailman3/mm/var/queue/shunt
> >
> > What should I do to get these messages processed?
>
>
> Assuming the issues that caused them to be shunted have been fixed, just
> run the command
> ```
> mailman unshunt
> ```
>
> > 3) Similar to above, some messages are in:
> >
> > /usr/local/mailman3/mm/var/queue/bad
> >
> > How can I get these messages processed?
>
>
> First examine the messages with
> ```
> mailman qfile /usr/local/mailman3/mm/var/queue/bad/<file>
> ```
> To be sure you want them. Also examine mailman.log for the times they
> were stored in the `bad` queue. You are looking for 'Skipping and
> preserving unparseable message: <file>' and/or 'SHUNTING FAILED,
> preserving original entry: <file> messages and you want to determine
> which runner preserved the message. There may be a 'whichq' entry in the
> message's metadata in which case, that's the answer.
>
> Once you have determined the queue, you can just move the .psv file from
> the bad queue to the appropriate queue, renaming it from .psv to .pck in
> the process.
>
>
> > 4 Modified the template for “list:admin:action:post” through Postorious and emails that arrived after that were put in the “shunt” queue. I traced this problem to POSTORIUS_TEMPLATE_BASE_URL. By default, it is set to "http://localhost:8000 <http://localhost:8000/>” which is where uwsgi is listening. However, uwsigi is listening for uwsgi:/ not http://. Should I change POSTORIUS_TEMPLATE_BASE_URL to “uwsgi://localhost:8000 <uwsgi://localhost:8000>” or keep http but make it something else?
>
>
> I know little about uwsgi. setting it to `uwsgi://localhost:8000` might
> work. You could also set it to the base URL that a user would use to
> access the web UI, e.g. `https://example.com` and that would presumably
> be proxied to uwsgi. For example, on the server for this list we have
> ```
> POSTORIUS_TEMPLATE_BASE_URL = 'https://lists.mailman3.org'
> ```
> which gets proxied to gunicorn, but the idea is the same.
Using the full external URL should work. Core doesn’t speak uwsgi protocol
so the uwsgi:// URL won’t actually work.
I am starting to think it was a very bad idea to use uwsgi in the official docs :(
--
thanks,
Abhilash Raj (maxking)
4 years, 1 month