Innstall and run Mailman 3 with Postfix in a Docker enviroment
by kwp.klein＠gmail.com
## Install Mailman 3
The idea is to install Mailman 3 in ad docker container. Also I want to install
Postfix in a container. All the mail which will send and receive should handled
in a docker container. The reason is, that on the server only applications in a
docker environment working. There is no MTA installed. On the server are
Discourse as forum, wikijs, portainer, watchtower, homepage, and the NGINX Proxy
Manager installed. At next I need Mailman 3.
I don't know how to install and run Postfix for Mailman in a docker environment.
Here is my first try. I can visit the site lists.example.com, but I can not set
the Admin password. I think the reason is, that the MTA not working.
I have a gitlab repo. If you like please look here:
https://gitlab.com/joklein/mailman
### Configure the docker-compose.yml file
```yml
version: '2'
services:
postfix:
image: mailu/postfix
restart: unless-stopped
container_name: postfix
volumes:
- /opt/mailman/postfix:/etc/postfix/main.cf.d
- /opt/mailman/core:/opt/mailman
ports:
- "25:25"
networks:
mailman:
ipv4_address: 172.19.199.6
services:
mailman-core:
image: maxking/mailman-core:0.3
restart: unless-stopped
stop_grace_period: 30s
container_name: mailman-core
hostname: mailman-core
volumes:
- /opt/mailman/core:/opt/mailman
links:
- database:database
depends_on:
- database
environment:
- DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb
- DATABASE_TYPE=postgres
- DATABASE_CLASS=mailman.database.postgresql.PostgreSQLDatabase
- SMTP_HOST=172.19.199.1
- HYPERKITTY_API_KEY=ZawB3NGBUgbmQcMGhZ0kYIW0WRNw8P9osEudsGiFZWv
networks:
mailman:
ipv4_address: 172.19.199.2
mailman-web:
image: maxking/mailman-web:0.3
container_name: mailman-web
restart: unless-stopped
hostname: mailman-web
ports:
- "8000:8000"
depends_on:
- database
links:
- mailman-core:mailman-core
- database:database
volumes:
- /opt/mailman/web:/opt/mailman-web-data
environment:
- DATABASE_TYPE=postgres
- DATABASE_URL=postgres://mailman:mailmanpass@database/mailmandb
- HYPERKITTY_API_KEY=ZawB3NGBUgbmQcMGhZ0kYIW0WRNw8P9osEudsGiFZWv
- UWSGI_STATIC_MAP=/static=/opt/mailman-web-data/static
- SERVE_FROM_DOMAIN=lists.example.com
- MAILMAN_ADMIN_USER=admin
- MAILMAN_ADMIN_EMAIL=lists(a)example.com
- SECRET_KEY=mailmangeheim
networks:
mailman:
ipv4_address: 172.19.199.3
database:
image: postgres:9.6-alpine
restart: unless-stopped
environment:
POSTGRES_DB: mailmandb
POSTGRES_USER: mailman
POSTGRES_PASSWORD: mailmanpass
volumes:
- /opt/mailman/database:/var/lib/postgresql/data
networks:
mailman:
ipv4_address: 172.19.199.4
networks:
mailman:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.19.199.0/24
```
#### Build the API Key für Hyperkitty and SECRET_KEY
You can build a key with `pwgen -s 128`
#### Make some directorys
```sh
mkdir -p /opt/mailman/core
mkdir -p /opt/mailman/web
mkdir -p /opt/mailman/mailman-web-data/static
```
#### Make a file in /opt/mailman/web/settings_local.py
```sh
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = '172.19.199.1'
EMAIL_PORT = 25
USE_SSL = False
DEFAULT_FROM_EMAIL = "<E-MAIL>"
SERVER_EMAIL = "<E-MAIL>"
```
Change the `<E-MAIL>` with your server address. For example
`mailman(a)lists.example.com`
So that Postfix also sends your e-mails, you create the file
`/opt/mailman/core/mailman-extra.cfg` with the following content:
```sh
[mta]
incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: 172.19.199.2
lmtp_port: 8024
smtp_host: 172.19.199.1
smtp_port: 25
configuration: /etc/postfix-mailman.cfg
```
Now you are ready to start your container:
```sh
sudo su -
cd /root/mailman-docker
docker-compose up -d
```
#### Check the mailman images
```sh
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f6990a2f808f maxking/mailman-web:0.3 "docker-entrypoint.s…" 55 seconds ago Up 52 seconds 8000/tcp, 8080/tcp mailman-web
a7fa6334b2b8 maxking/mailman-core:0.3 "docker-entrypoint.s…" 58 seconds ago Up 54 seconds 8001/tcp, 8024/tcp mailman-core
e8812dc42663 postgres:9.6-alpine "docker-entrypoint.s…" About a minute ago Up 57 seconds 5432/tcp docker-mailman_database_1
```
#### Check the conection
```sh
curl http://172.19.199.3:8000/postorius/lists/
```
4 years, 11 months
Re: Hyperkitty "All Threads" results in error
by Mark Dadgar
Aaaand I spoke too soon. I am now seeing the same error when I click the All Threads button in hyperkitty. Not every time, because that would be too easy, but often.
I really only have one high-volume list with archives, so I can’t tell if it’s list-specific.
I ran the mailman-post-update script after the last update, which I believe executes all the commands suggested below.
Here’s the crash trace.
Any thoughts?
- Mark
Internal Server Error: /archives/list/trackjunkies(a)pdc-racing.net <mailto:archives/list/trackjunkies@pdc-racing.net>/latest
VariableDoesNotExist at /archives/list/trackjunkies(a)pdc-racing.net <mailto:archives/list/trackjunkies@pdc-racing.net>/latest
Failed lookup for key [sender] in None
Request Method: GET
Request URL: https://mail.pdc-racing.net/archives/list/trackjunkies@pdc-racing.net/latest <https://mail.pdc-racing.net/archives/list/trackjunkies@pdc-racing.net/latest>
Django Version: 3.0.12
Python Executable: /opt/mailman/mm/venv/bin/python
Python Version: 3.8.5
Python Path: ['/opt/mailman/mm', '/opt/mailman/mm/', '/opt/mailman/mm/bin', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/opt/mailman/mm/venv/lib/python3.8/site-packages']
Server time: Mon, 1 Mar 2021 23:25:02 -0800
Installed Applications:
('hyperkitty',
'postorius',
'django_mailman3',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'django_gravatar',
'compressor',
'haystack',
'django_extensions',
'django_q',
'allauth',
'allauth.account',
'allauth.socialaccount')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'django_mailman3.middleware.TimezoneMiddleware',
'postorius.middleware.PostoriusMiddleware')
Traceback (most recent call last):
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 828, in _resolve_lookup
current = current[bit]
During handling of the above exception ('NoneType' object is not subscriptable), another exception occurred:
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 836, in _resolve_lookup
current = getattr(current, bit)
During handling of the above exception ('NoneType' object has no attribute 'sender'), another exception occurred:
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 842, in _resolve_lookup
current = current[int(bit)]
During handling of the above exception (invalid literal for int() with base 10: 'sender'), another exception occurred:
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 34, in inner
response = get_response(request)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/hyperkitty/lib/view_helpers.py", line 134, in inner
return func(request, *args, **kwargs)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/hyperkitty/views/mlist.py", line 115, in archives
return _thread_list(request, mlist, threads, extra_context=extra_context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/hyperkitty/views/mlist.py", line 144, in _thread_list
return render(request, template_name, context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/shortcuts.py", line 19, in render
content = loader.render_to_string(template_name, context, request, using=using)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/loader.py", line 62, in render_to_string
return template.render(context, request)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/backends/django.py", line 61, in render
return self.template.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 171, in render
return self._render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 150, in render
return compiled_parent._render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 62, in render
result = block.nodelist.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/defaulttags.py", line 209, in render
nodelist.append(node.render_annotated(context))
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/loader_tags.py", line 188, in render
return template.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 173, in render
return self._render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 163, in _render
return self.nodelist.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/defaulttags.py", line 513, in render
return self.nodelist.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 936, in render
bit = node.render_annotated(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 903, in render_annotated
return self.render(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/templatetags/i18n.py", line 132, in render
context.update({var: val.resolve(context) for var, val in self.extra_context.items()})
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/templatetags/i18n.py", line 132, in <dictcomp>
context.update({var: val.resolve(context) for var, val in self.extra_context.items()})
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 691, in resolve
arg_vals.append(arg.resolve(context))
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 795, in resolve
value = self._resolve_lookup(context)
File "/opt/mailman/mm/venv/lib/python3.8/site-packages/django/template/base.py", line 847, in _resolve_lookup
raise VariableDoesNotExist("Failed lookup for key "
Exception Type: VariableDoesNotExist at /archives/list/trackjunkies(a)pdc-racing.net <mailto:archives/list/trackjunkies@pdc-racing.net>/latest
Exception Value: Failed lookup for key [sender] in None
Request information:
USER: mark
GET: No GET data
POST: No POST data
FILES: No FILES data
On Feb 17, 2021, at 5:34 PM, Mark Dadgar <mark(a)pdc-racing.net> wrote:
>
> On Oct 14, 2020, at 4:37 PM, Mark Sapiro <mark(a)msapiro.net> wrote:
>> On 10/14/20 11:12 AM, Mark Dadgar wrote:
>>>> On Oct 4, 2020, at 10:34 AM, Mark Dadgar <mark(a)pdc-racing.net> wrote:
>>>>
>>>> So I have been wrestling with this problem for a while now. Most times, but not every time, when someone clicks on the All Threads link in hyperkitty, a 500 error is generated.
>>
>>
>> Does this happen on all lists, only some lists or only one list?
>>
>>
>>>> I googled it, obviously, and at first I thought it was related to this issue, but that has not proven to be the case:
>>>>
>>>> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/thread/…
>>
>> What did you google? This query gets a lot of stuff with some hints, but
>> nothing definitive.
>>
>> django render raises VariableDoesNotExist("Failed lookup for key "
>>
>> One thing I noted at
>> <https://askbot.org/en/question/3095/django-error-variabledoesnotexist-faile…>
>> was
>>
>> models out of date - run python manage.py syncdb and python manage.py
>> migrate
>>
>> You could try that. Note that your django admin command might be
>> something other than `python manage.py`. Also note syncdb has been
>> removed from Django >=1.9. Currently, the above becomes `makemigrations`
>> followed by `migrate`.
>>
>> I also suggest running the django admin commands
>>
>> `collectstatic --clear --noinput`
>> and
>> `compress`
>
>
> My re-install seems to have fixed this issue as well. BONUS!
>
> - Mark
> -----
> mark(a)pdc-racing.net | 408-348-2878
>
>
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
- Mark
-----
mark(a)pdc-racing.net | 408-348-2878
4 years, 10 months
Re: how to integrate subscription into static jekyll website
by Stephen J. Turnbull
Hagen Bauer writes:
> Has anybody tried to hide the direct api access behind a nginx reverse
> proxy url?
Sounds straightforward.
I doubt anybody's done it, though, because you still face the problem
that anybody with that access is a site manager, with unclear limits
on their power outside of Mailman. (Shouldn't be much they can do
outside of Mailman, but I wouldn't bet my host on it.) In many cases,
the real site manager is also the host admin, so they have sufficient
access to a shell account. There's no need for her to have a proxy;
why make it available to people who are supposed to have less power?
Note I'm *not* saying this *never* makes sense. I'm just saying that
in a lot of cases it won't buy you very much because Mailman isn't set
up to provide granular authorization of REST endpoints.
> on the Webseite I would have a url like this
>
> mailmanserver.tld/subcsribe/user(a)mail.com
>
> And within the nginx proxy configuration on the server this would
> translate to
>
> localhost:80xx/"mailman-restapi-with-credentials"user(a)mail.com
First of all, the address to subscribe is a variable, so at a bare
minimum your nice static site has to provide a form, which isn't very
static. I guess you could put the URL to the nginx proxy as the
action URL in the form, but then the proxy needs to deal with a
form-encoded query string to translate to the appropriate REST
endpoint.
I suppose you could also have the user type the URL including the
email address into the browser address bar, but Abhilash's idea of
providing a mailto:subscribe@mailmanserver.tld URL seems more
convenient. The mailto URL also solves the problem of confirming
ownership of the mailbox, since that logic is built in to the
subscribe-by-mail code.
Second, if your proxy is configured very carefully, you might be able
to restrict this to only the exact operations you want to expose, here
subscription. However, although the email interface implements
verification by providing a one-time token that must be read from the
mailbox of the address (and so proves ownership of the mailbox), I
don't think the REST API implements operations like that. IIRC that
is implemented in Postorius. (Abhilash?) If my memory is correct,
you would also need to implement that, and that's definitely not static.
> So the public does not see the credentials. And if my nginx host
> config is hacked I have a problem anyway.
I can't speak for Abhilash, but what would worry me is not that nginx
is already hacked, but rather that if not properly configured, it
might be possible for someone to access REST endpoints you don't want
exposed through this interface, and maybe even to other parts of the
webserver. To give a simple example, consider a variant on the
ancient Apache traversal bug:
mailmanserver.tld/subscribe/../unsubscribe/user(a)mail.com
Probably nginx and/or Mailman and/or your proposed proxy config are
already capable of stopping this particular well-known attack, but
undoubtedly there are more sophisticated attacks that I don't know
about.
7 years, 2 months
Re: Problems after update to mailman 3.3.7
by Guillermo Hernandez (Oldno7)
El 6/12/22 a las 15:59, Danil Smirnov escribió:
> Hi Guillermo,
>
> We've faced the same issue and already reported it:
> https://gitlab.com/mailman/mailman/-/issues/1044
>
I see. Thanks, Danil, for the confirmation that I'm not alone in this..
> I would not recommend anyone using MySQL/MariaDB database engine upgrade to
> the latest version yet.
Too late for that... Doy you know what components could I downgrade to
avoid this? SQLAlchemy? Which versions of which packages could I
downgrade to work prpoerly?
Again, thanks a lot
> With my best regards,
> Danil Smirnov
> Mailman3.com
>
> On Tue, Dec 6, 2022 at 4:32 PM Guillermo Hernandez (Oldno7) via
> Mailman-users<mailman-users(a)mailman3.org> wrote:
>
>> I've getting the most strange behaviour after update to last releases.
>> In one of my mailman3 servers, the messages doesnt reach for some lists
>> (the five last ones)
>>
>> the update have not showed any error but in the var/data for the lmtp
>> and vmap postfix files mailman does not create aliases for that last 5
>> lists. Any message addressed to them is returned with a "no such user"
>> error.
>>
>> If I create a new list the system or restart mailman runners it shows
>> this error:
>>
>> Generating MTA alias maps
>> /usr/local/lib/python3.9/site-packages/pymysql/connections.py:799:
>> UserWarning: Previous unbuffered result was left incomplete
>> warnings.warn("Previous unbuffered result was left incomplete")
>>
>> I did a CHECK TABLE on mailinglists table but all seems ok.
>>
>> I tried reinstalling PyMySQL, but has no effect. (well, I tried
>> reinstalling all)
>>
>> It seems to me that something is preventing creation of that aliases, an
>> then the that last five lists are unreachable.
>>
>> I have no clue of what to do now and I'm driving crazy.
>>
>> Some direction to look for will be much apreciated.
>>
>> Thanks in advance.
>>
>> P.S.: I can access via postorius/hyperkitty to the archives of the lists
>> that are unreacheable.
>>
>>
>> --
>>
>> ___________________________________________
>> Mailman's content filtering has removed the
>> following MIME parts from this message.
>>
>> Content-Type: image/png
>> Name: firma-GHP-emails.png
>>
>> Replaced multipart/alternative part with first alternative.
>> _______________________________________________
>> Mailman-users mailing list --mailman-users(a)mailman3.org
>> To unsubscribe send an email tomailman-users-leave(a)mailman3.org
>> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>> Archived at:
>> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>>
>> This message sent todanil(a)smirnov.la
>>
> _______________________________________________
> Mailman-users mailing list --mailman-users(a)mailman3.org
> To unsubscribe send an email tomailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at:https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/mess…
>
> This message sent toguillermo(a)querysoft.es
>
--
___________________________________________
Mailman's content filtering has removed the
following MIME parts from this message.
Content-Type: image/png
Name: firma-GHP-emails.png
Replaced multipart/alternative part with first alternative.
3 years, 1 month
Re: Mailman, etc. upgrade woes and persistent bugs
by Stephen J. Turnbull
No answer required, I think I understand the situation pretty well
now. But if I'm missing something, I would very much appreciate
criticism.
Andreas Barth writes:
> Well, currently these are exclusive-states, I'd rather see them as
> or-able-states.
Right, I understand that. I don't understand *why*.
There are real costs to making them or-able. There are constraints:
'enabled' shouldn't (mustn't?) be or-able with any 'disabled-by-foo'
state. Such invariants are finicky to code and tricky to maintain,
compared to a simple "mutually exclusive".
I guess you want to allow the or of any subset of disabled states, but
presumably then they need to individually resettable in order for the
admin to clear their disable without overriding a subscriber's
preference or a bounce record. That set of controls is more complex,
and requires additional mental effort on the part of anyone using
them. If there are restrictions on which states can be or-ed, that's
more coding and maintenance cost.
Then there's the question whether the subscriber can override the admin's
setting. If they can, why does the admin have this power in the first
place?
> > > - "disabled by member" (could be set by an admin as well of course)
> >
> > Currently this is determined by whether the logged in user is the
> > subscriber or not. It's an interesting idea to allow it to be set by
> > an admin. Then we could have the semantics that the only transition
> > allowed from this state would be to enabled, so that admins would know
> > the user had intentionally disabled and know not to enable without
> > permission.
>
> Exactly.
But if the admin should respect the DBS (disabled by subscriber)
setting, DBS|DBA (disabled by admin) is logically equivalent to DBS.
Are there use cases where the admin does not want to repect the
subscriber's choice, and enables it anyway? What are they?
> Or while importing from another server, and there are people who
> have receiving mails disabled, this is the state that could be used
> for it.
Why wouldn't they get the exact state from the original server? Or do
you mean from a non-Mailman server?
> I expect that state to be the most-often used when setting a state
> manually.
>
> For "who set that", I would consider having some kind of history as
> useful (and displaying "last set by user / $person" to the admin but
> not the user).
Why not to the subscriber? As I explain below, the subscriber at
least needs to be able to distinguish between DBA and DBB (disabled by
bounce).
Keeping history of who is additional complexity, consumes a minor
amount of resources, and is insufficient to the purpose. Assuming
that there are reasons for admins to disable/enable other than as a
courtesy to a subscriber, that reason is presumably important. Not only
will the subscriber not know, but one admin may forget, and there may be
multiple admins.[1]
I still don't see any reason for an admin to disable delivery other
than as a courtesy to the subscriber or to the bounce processor.
> > > - "disabled for administrative reasons" (only by admin but user
> > > should be able to see it)
> >
> > I don't see why a user would need to see the difference. Unless
> > you're suggesting that the user would not be able to reenable without
> > asking the admin? What is the use case for this?
>
> I'm suggesting exactly that, yes. Why? Well, worked with user for too
> long. ;)
>
> E.g. consider that seeing archives works only for subscribers. And the
> subscriber in question complains about too many mails but still
> re-enables getting mails. This is basically "an admin has blocked you
> from more mails but not unsubscribed you". Might be an corner case,
> but still. Might also be too unimportant.
This is possible. If it occurs less than once in a million years,
it's "too unimportant" to even think about.[2] Have you actually done
this, or know anybody who has?
Is it really different from "at request of subscriber"? Practically,
the only difference I can see is if such a subscriber should be
prevented from overriding DBA. Are you suggesting that?
> > > - "bounced" (user and admin could reset it but not set it)
> >
> > Reenable but not set "bounced" is the current situation.
>
> Except if someone sets the state to "disabled" and then undo this, the
> state is automatically cleared.
This is possible, but it's not clear what harm would be done. In most
cases, bounces are intermittent and clear themselves up, in which case
this is the right thing. But if the subscriber bounces again, they'll
get disabled again, only cost is a slight waste of resources.
Here's how I would assess the issues. In the descriptions below, I am
assuming that each state is the only "active" state.
1. DBS is probably a "vacation" or "post-only" setting. For that
reason it should be re-enabled only by the subscriber (or by an
admin at the request of the subscriber).
2. DBB is a "warning" and a minor convenience to the email system
(slightly reducing traffic to mailboxes that are expected to fail,
and perhaps avoiding some reputational damage to the list).
Re-enabling much of the time causes no damage, as bounce disables
are typically due to things like out of disk space, and are likely
fixed by the time the subscription is reenabled. Most of the rest
will be zombie mailboxes, which will rarely be reenabled, will
quickly be redisabled by bounce, and at worst cause mild
inconvenience. Finally, events like a Mailman upgrade and the
April 2014 DMARC fiasco will be pretty evident, and the damage
there has already been done, reenabling actually fixes it
(although it may not be effective if the systemic problem hasn't
been resolved).
3. DBA is interesting mostly to *subscribers*, who need to
distinguish DBA (which the subscriber can reenable freely) from
DBB (which warns the subscriber to get in touch with their
postmaster). It's comforting to subscribers who may be sure they
never disabled their subscription; otherwise, it's semantically
identical to DBS as far as I can see.
Am I missing something?
It seems to me that given 1-3, assuming a 4-state design:
0. It's never unreasonable to re-enable, except DBS.
1. In the DBA state, changing to either DBS or DBB causes no
problems. (Changing to DBB is unlikely, since no mail is being
sent. However, it's possible, due to the race between Postorius
and mail in transit, and due to spam and other spoofing of the
list domain.)
2. In either of the other two states, we should never change to DBA.
3. In either of the other two states, if the other event occurs, you
would want the state to be DBS|DBB. I contend that if we
substitute DBS, little harm is done. If the subscriber reenables, it's
on them to fix the bounce problem if it reoccurs in any case.
Since the subscriber has deliberately set DBS, the admin should not
reenable without permission of the subscriber (I think we're mostly in
agreement on that).
I think this 4-state design is much easier to design, validate, and
maintain than the 16-state design you propose. If you're really
worried about DBS|DBB, then we could have a 5-state design where
enabling from DBS|DBB generates a warning that there may still be a
bounce problem on the subscription. Instead, it could leave the
subscription in state DBB, but that seems to me to be obnoxious, since
there's little likelihood of harm from reenabling, and in many cases
the bounce problem will have resolved itself in the meantime.
Regards,
Steve
Footnotes:
[1] There's also the question of GDPR obligations if personal reasons
(such as "violated code of conduct") are in the history.
[2] I'm not sure where we'd actually draw the line, probably on the
order of "once a year per thousand subscribers", that being every week
for say a campus-wide list at Ohio State University with 55,000 students.
4 years, 10 months
Re: Getting connection refused on postfix trying to connect to mailman-core:lmtp
by Dmitry Makovey
On 10/05/2017 07:20 PM, Abhilash Raj wrote:
> On Thu, Oct 5, 2017, at 05:36 PM, Dmitry Makovey wrote:
>> On 10/05/2017 03:20 PM, Abhilash Raj wrote:
>>> On Thu, Oct 5, 2017, at 03:08 PM, Dmitry Makovey wrote:
>>>>
>>>> I've got a setup where postfix runs inside one VM(container) and mailman
>>>> runs inside another one (maxking containers). I've wired everything
>>>> according to docs yet I'm getting:
>>>>
>>>> postfix/lmtp[266]: 66A72800A87: to=<somelist(a)lists.here.stanford.edu>,
>>>> relay=none, delay=0.5, delays=0.48/0.01/0/0, dsn=4.4.1, status=deferred
>>>> (connect to mailman-01.stanford.edu[1.2.3.4]:8024: Connection refused)
>
> I found this in your settings:
>
> [mta] lmtp_host: mailman-01.stanford.edu
> [mta] lmtp_port: 8024
>
>
> And the log message above.
>
> I believe that the LMTP runner died because it wasn't able to bind to
> `mailman-01.stanford.edu`, which I am assuming is the hostname assigned
> to the host running these containers.
>
> `MM_HOSTNAME` env variable in the docker containers should be something
> that the process inside mailman-core container can bind to and can be
> reached by postfix (which can run either on host or on another
> container). (Now that I read it myself, I agree that the name of the
> variable sounds not-so-intuitive.)
thank you so much for the hints! I've changed docker-compose to include
MM_HOSTNAME variable *and* made sure that for the mailman-core I've got:
services:
mailman-core:
hostname: mailman-01
domainname: stanford.edu
...
environment:
...
MM_HOSTNAME: mailman-01.stanford.edu
that solved the issue while keeping the mentioned mailman config. Is
that what you've had in mind? I was looking for a quick hack, but still
would like to find out proper solution if that isn't the one.
> The default configuration (and docker-compose.yaml) sets this value to
> the IP Address of the container (172.19.199.2), which is reachable from
> the host. If you set this value to whatever IP the mailman-core is
> assigned and re-create the containers (or just re-start and run `mailman
> aliases` in mailman-core to re-generate transport_maps), it would work
> out.
if I understand above correctly that means semi-manual mangling of
postfix aliases file which I'd rather not do. Using above technique
mailman-core does generate proper aliases while binding to the
appropriate IP. kind of icky but seems to work.
>
> Also, I see your docker-compose.yaml configuration (`MAILMAN_CORE_IP:
> 172.19.199.2`) is not consistent with your output of `mailman
> conf`([webservice] hostname: 172.19.199.5`). I am not sure how that
> happened though, just wanted to point it out. This *might* cause the
> rest runner to die to and thus Postorius/HK wouldn't work.
right, some of the IPs keep on popping at me and I've got to look for
them - I intentionally removed direct IP assignment in
docker-compose.yaml yet I get the feeling that someplace there's another
IP assignement I've missed.
> Hope that was helpful!
very much so! thanks for your quick responses - provided my existing
deadlines I value them that much more :)
--
Sr System and DevOps Engineer SoM IRT
8 years, 3 months
Re: where to configure max_days_to_hold on MM3 GUI interface?
by Abhilash Raj
On Sat, Feb 1, 2020, at 5:27 AM, Shashikanth Komandoor wrote:
> Hello Mark,
>
> Thanks for helping me for moving ahead in my project.
>
> Troubling you again, I would like to put another request at you.
>
> As per your suggestion, I am trying to configure my list
> configuration through REST API for specially the parameter like
> max_days_to_hold as it is not available through interface and I am
> following the official links of mailman 3 i.e
> https://mailman.readthedocs.io/en/latest/src/mailman/rest/docs/listconf.html
>
> But I am getting "name is not defined" sort of errors as below:
>
>
>
>
>
> *>>> dump_json('http://localhost:9001/3.0/lists/
> <http://localhost:9001/3.0/lists/>','ant(a)example.com/config
> <http://ant@example.com/config>',dict(display_name='My
> List'),'PATCH')Traceback (most recent call last): File "<stdin>", line 1,
> in <module>NameError: name 'dump_json' is not defined>>>*
>
> Actually I am new to REST API but hope this is the only issue that
> is preventing me to go ahead.
>
> Could you please help me by saying how should I handle this so that
> I should be able to configure *max_days_to_hold* of a list or multiple ?
>
dump_json is basically a Python function which sends the request to the URL you add as the first parameter (GET, if any data isn't specified and uses POST if data is specified or explicitly define the PATCH or method like you have above) and prints the json response.
You can easily do the same using curl with the right authorization headers, with something like:
$ curl -u restadmin:restpass -X PATCH \
http://localhost:8001/3.1/lists/somelist.example.com/config -d display_name='Patched Name'
Note that "restadmin:restpass" is the username password that you have set in your mailman.cfg.
If you do want to use Python, you can just use a regular PATCH request using requests or standard library urllib3.
Finally, dump_json is documented here: https://mailman.readthedocs.io/en/latest/src/mailman/docs/documentation.htm…
> On Wed, Jan 29, 2020 at 11:20 PM Mark Sapiro <mark(a)msapiro.net> wrote:
>
> > On 1/28/20 8:42 PM, Shashikanth Komandoor wrote:
> > >
> > > But I could see that parameter value configurable on MM2 GUI
> > > interface in General Options but I don't see the same on the MM3 GUI
> > > interface.
> >
> >
> > There are many core settings not (yet) exposed in Postorius.
> > max_days_to_hold. is one of them. It can be set via `mailman shell` or
> > via the REST API
> > <
> > https://mailman.readthedocs.io/en/latest/src/mailman/rest/docs/rest.html#re…
> > >.
> >
> > --
> > Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> > San Francisco Bay Area, California better use your sense - B. Dylan
> > _______________________________________________
> > Mailman-users mailing list -- mailman-users(a)mailman3.org
> > To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >
>
>
> --
> Thanks & Regards,
> Shashi Kanth.K
> 9052671936
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>
--
thanks,
Abhilash Raj (maxking)
5 years, 11 months
Re: mailman-web migrate fails with sqlite3.IntegrityError: UNIQUE constraint failed: account_emailaddress.email
by Stefan Bauer
Dear Mark,
thank you for your time.
After patching your script according to
https://github.com/maxking/docker-mailman/issues/729#issuecomment-2447542238
It fails with
/opt/mailman/venv/data# /usr/local/bin/UC_fix
/opt/mailman/venv/data/mailman3web.db
Traceback (most recent call last):
File "/usr/local/bin/UC_fix", line 77, in <module>
c.execute("delete from account_emailaddress where email = %s;",
sqlite3.OperationalError: near "%": syntax error
for reference:
67 for id, emails in d.items():
68 ems = find(emails)
69 if ems:
70 em1, em2 = ems
71 if em1.lower() == em1:
72 c.execute("delete from account_emailaddress where email =
%s;",
73 [em2])
74 c.execute("""update account_emailaddress set "primary" = 't'
75 where email = %s;""", [em1])
76 elif em2.lower() == em2:
77 c.execute("delete from account_emailaddress where email =
%s;",
78 [em1])
79 c.execute("""update account_emailaddress set "primary" = 't'
80 where email = %s;""", [em2])
81 else:
82 print(f'No LC email for {ems}')
However the sql connection itself works, and the select also reports
addresses:
sqlite> select email, user_id from account_emailaddress;
...
John.Doe(a)my.domain|85
john.doe(a)my.domain|85
.....
Mike.Doe(a)my.domain|96
mike.doe(a)my.domain|96
....
Is there something wrong on my side?
Thank you.
Stefan
Am Mo., 3. März 2025 um 18:40 Uhr schrieb Mark Sapiro <mark(a)msapiro.net>:
> On 3/3/25 07:52, Stefan Bauer via Mailman-users wrote:
> >
> > trying to update mailman3 to lastest version, we fail during
> >
> > mailman-web migrate with:
> >
> > Operations to perform:
> > Apply all migrations: account, admin, auth, contenttypes,
> django_mailman3,
> > django_q, hyperkitty, postorius, sessions, sites, socialaccount
> > Running migrations:
> > Applying account.0006_emailaddress_lower...Traceback (most recent call
> > last):
> > File
> >
> "/opt/mailman/venv/lib/python3.10/site-packages/django/db/backends/utils.py",
> > line 89, in _execute
> > return self.cursor.execute(sql, params)
> > File
> >
> "/opt/mailman/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py",
> > line 328, in execute
> > return super().execute(query, params)
> > sqlite3.IntegrityError: UNIQUE constraint failed:
> account_emailaddress.email
>
>
> This issue is due to changes in Django allauth to prevent creating
> multiple addresses differing only in case. The problem is if you have
> such multiple addresses in your account_emailaddress table, this
> migration fails. There is a script at
> https://www.msapiro.net/scripts/UC_fix that will fix this. You may need
> to adjust the shebang and replace psycopg2 with sqlite3.
>
> See https://github.com/pennersr/django-allauth/issues/3019 perhaps
> starting at
>
> https://github.com/pennersr/django-allauth/issues/3019#issuecomment-1748028…
>
> > We found a workaround to fake the migration with
> >
> > migrate --fake
>
> You should fix the issue with the DB and run the migrations.
>
> > Could this be some duplicate users with same mail-address?
>
> This is not related to Mailman users. It is Django users.
>
> --
> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> San Francisco Bay Area, California better use your sense - B. Dylan
>
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at:
> https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>
> This message sent to cubewerk(a)gmail.com
>
10 months
Re: Getting rid of held messages from the command line
by Mark Sapiro
On 03/16/2018 09:01 AM, Aymeric Mansoux wrote:
>
> However, while I was testing this happily on a list that was not affected with the JSON error, I finally tested discard_held_messages() on one of the problematic list, and I get the exact same problem as when trying to view the moderation queue API:
>
>>>> discard_held_messages_queue('xxx(a)xxx.xxx')
> Traceback (most recent call last):
> File "<console>", line 1, in <module>
> File "<console>", line 6, in discard_held_messages_queue
> File "/usr/local/lib/python3.5/dist-packages/mailman/app/moderator.py", line 97, in handle_message
> key, msgdata = requestdb.get_request(id)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/model/requests.py", line 120, in get_request
> result.data_hash, expunge=False)
> File "/usr/local/lib/python3.5/dist-packages/mailman/database/transaction.py", line 85, in wrapper
> return function(args[0], config.db.store, *args[1:], **kws)
> File "/usr/local/lib/python3.5/dist-packages/mailman/model/pending.py", line 138, in confirm
> value = json.loads(keyvalue.value)
> File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib/python3.5/json/decoder.py", line 355, in raw_decode
> obj, end = self.scan_once(s, idx)
> json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 1 (char 0)
>
> With that said, even though I cannot pass the discard action, I can export the queue without problem, so I will send you off-list a sample, because I am not sure what I should be looking for.
Unfortunately, that didn't help. I was able to post all the messages in
your sample to a test list and even with all the messages held, I could
view them and discard them in postorius.
This may not be too surprising as looking at the traceback, the
exception is thrown in trying to delete the pending confirmation from
the pending database. The issue is when a message is held an entry is
made in the pending database with a confirmation token that can be used
for example to approve or discard the message by email.
There is an issue that affects MySQL databases only in that prior to the
merge of <https://gitlab.com/mailman/mailman/merge_requests/333> values
in the pendedkeyvalue table could be truncated (or exceptions thrown
depending on MySQL settings). That appears to be the issue here.
The fix was only merged in January and is not in Mailman 3.1.1.
> Finally, is there another way to get rid of these messages? Are there consequences if I would use message_store.delete_message(message_id) instead of handle_message(m, req.id, Action.discard)?
As I said, I think the issue is with the pendedkeyvalue table, not with
the message itself. The table contains entries which are pickled
versions of various things. A big issue is rule_misses. For one of your
messages, the rule_misses list is
['dmarc-mitigation', 'no-senders', 'approved', 'emergency', 'loop',
'banned-address', 'member-moderation']
this gets pickled into
b'\x80\x03]q\x00(X\x10\x00\x00\x00dmarc-mitigationq\x01X\n\x00\x00\x00no-sendersq\x02X\x08\x00\x00\x00approvedq\x03X\t\x00\x00\x00emergencyq\x04X\x04\x00\x00\x00loopq\x05X\x0e\x00\x00\x00banned-addressq\x06X\x11\x00\x00\x00member-moderationq\x07e.'
which ultimately gets stored in the pendedkeyvalue table as a value for
'_pck_rule_misses' which is
'"\u0080\u0003]q\u0000(X\u0010\u0000\u0000\u0000dmarc-mitigationq\u0001X\n\u0000\u0000\u0000no-sendersq\u0002X\b\u0000\u0000\u0000approvedq\u0003X\t\u0000\u0000\u0000emergencyq\u0004X\u0004\u0000\u0000\u0000loopq\u0005X\u000e\u0000\u0000\u0000banned-addressq\u0006X\u0011\u0000\u0000\u0000member-moderationq\u0007e."'
Which is longer than the VARCHAR(255) MySQL field for that value prior
to <https://gitlab.com/mailman/mailman/merge_requests/333>.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
7 years, 9 months
Re: UnicodeEncodeError: 'ascii' codec can't encode character
by Stephen J. Turnbull
Marvin Gülker writes:
> Am 10. Dezember 2019 um 17:09 Uhr +0900 schrieb Stephen J. Turnbull:
> > You're the "it's 2019, let's use SMTP UTF8 everywhere" guy, right?
>
> No, I'm not that guy.
I'm sorry about that. I knew I didn't know for sure, but not being
able to get at the moderation queue seemed urgent enough to be worth
the chance of being wrong.
> I have not used it, but it turns out that it is enabled by
> default. I can turn it off and see if that makes any difference in
> the future. Testing with telnet, Postfix indeed announced "250
> SMTPUTF8" on EHLO.
>
> > If you're right about the umlaut being the trigger, the first
> > thing I'd look at is a feature-negotiation problem in the MTA
> > delivering to Mailman. I'm pretty sure our LMTP does not offer
> > the SMTP UTF8 feature.
>
> In that case the problem should go away with disabling SMTPUTF8, I
> suppose?
Well, since it's predicated on a bug in Postfix, I wouldn't bet on it,
especially since Postfix probably dominates Mailman 3 installations.
But it's cheap to test, and since you don't explicitly want SMTP UTF8
at this point, little harm in turning it off. *If* there is a bug,
that will prevent the problem in the future, but it wouldn't fix the
problem in the held messages queue.
> > I would guess this is in the shunt queue. If there's only one
> > file there, you can just delete it.
This was a bad guess. I'm not sure why it isn't in the shunt queue
(it shouldn't have been able to escape into the main rule chains with
that bad breakage in the header), but Mailman doesn't try to moderate
shunted messages, that's the whole point of the shunt queue -- they're
out of the way.
> There was a whole bunch of files in that directory. I've taken a look
> several of them, and as they were all spam, I've taken the liberty to
> delete them all (this list is so low volume that I know all the senders
> personally anyway).
A tiny bit of good came of it, at least.
> However, even after restarting mailman, the 500 error persists when
> visiting the held messages page in postorius.
>
> I've also looked into all the other queue directories, they were all
> empty.
I guess it's in the MessageStore, then, which is by default in
/var/tmp/mailman/messages/. (Defined in schema.cfg.) I guess it's a
standard qfile that you can examine with "mailman qfiles
/path/to/file". (I've never actually looked at a "raw" held message
file in Mailman 3, haven't had a failure of this type.) If not, you
can unpickle it with Python (if that makes no sense to you, ask; I'm
running out of steam and need sleep).
> > Assuming you're right about your name being the trigger, U+FFFD
> > REPLACEMENT CHARACTER is not in your name.
I'm starting to wonder about this. It's still possible, but if you
don't deliberately use SMTP UTF8 then your mail client probably
doesn't either, and equally possibly it's a different message causing
the problem (something spammy).
6 years, 1 month