Re: MM3 Docker
by Odhiambo Washington
On Sun, 6 Jan 2019 at 23:39, Mark Sapiro <mark(a)msapiro.net> wrote:
> On 1/6/19 8:47 AM, Odhiambo Washington wrote:
> >
> > If you could include the missing steps that you left out then I could
> > possibly be able to follow the steps :-)
>
>
> Please help me understand what steps are missing and I will try to
> provide the information.
Okay.
At the installation stage, you say the following:
"I already had some things set up in /opt/mailman including a git
subdirectory containing clones of the GitLab mailman, mailmanclient,
mailman-hyperkitty, hyperkitty, django-mailman3 and
postorius projects. On the third server I have added mailman-suite
because I use the settings.py from that project as the basis for mine."
Please assume that the reader is a newbie to python3 virtualenv and give the
details of the commands they need to execute to (a) clone all the projects
needed
and (b) create the virtualenv.
The next steps are:
mkdir /opt/mailman/mm
Then you say "I then created a /opt/mailman/mm directory and within that a
Python 3.6 virtualenv"
What happens here is something that needs clarification, because you are
specific about "python3.6".
I have googled and realized that one has to be specific about the python
version they want to use in their virtualenv
as shown here ->
https://stackoverflow.com/questions/45293436/how-to-specify-python-version-…
exact command to execute to create the /opt/mailman/mm/venv
exact command to execute to activate virtualenv
From there, I think the newbie can follow and fumble till they come up with
a working MM3 site.
Come to think of it, your steps kind of look clearer than most that I have
encountered.
I do understand that there are only a few people working on the MM3
development, which I think is
the reason there still isn't a document similar to the ones included with
MM2.1x source under doc/
However, if your steps here are polished a little, they can possibly become
one such doc.
I think MM3 docs would come with three options - virtualenv option, docker
option and package option.
Installing from source seems like something not easily doable.
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)
7 years, 10 months
list archives bleeding across domains.
by Hugh Esco
I launched a mailman installation a couple of days ago, `pip list and
`cat /etc/os-release` output reproduced below. It hosts six domains
and a handful of lists for most of them.
My issue is that lists archives are being linked across domain names
for annonymous user, and I fear probably for users who are logged in as
well.
I get it that one domain's news list is supposed to be public, but does
it have to be listed on the page for each domain? Can't it be listed
for anonymous users ONLY IF they browse to its domain?
My settings_local.py includes:
FILTER_VHOST = True
SITE_ID = 0
I have another reason (MAILMAN_ARCHIVER_KEY is working) to believe that
my settings_local.py file is being read into the configuration.
Any advice?
All help is appreciated.
Thanks,
-- Hugh Esco
((mm3) ) list@lists02:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.5 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
list@lists02:~$ . /opt/local/mm3/bin/activate
((mm3) ) list@lists02:~$ mailman --version
GNU Mailman 3.3.9 (Tom Sawyer)
((mm3) ) list@lists02:~$ mailman-web --version
4.2.30
((mm3) ) list@lists02:~$ pip list
Package Version
------------------- -----------
aiosmtpd 1.4.6
alembic 1.20.0
asgiref 3.12.1
atpublic 9.0.0
attrs 26.1.0
authheaders 0.16.3
authres 1.2.0
certifi 2026.7.22
cffi 2.1.1
charset-normalizer 3.5.2
click 8.5.0
comrak 0.0.16
cryptography 50.0.2
defusedxml 0.7.1
Django 4.2.30
django-allauth 65.19.7
django-appconf 1.2.0
django_compressor 4.6.0
django-extensions 4.1
django-gravatar2 1.4.5
django-haystack 3.4.0
django-mailman3 1.3.15
django-picklefield 3.4.0
django-q2 1.10.0
djangorestframework 3.17.2
dkimpy 1.1.8
dnspython 2.9.0
docutils 0.23
falcon 4.4.0
flufl.bounce 6.0.0
flufl.i18n 8.0.0
flufl.lock 10.0.0
greenlet 3.5.6
gunicorn 26.2.0
HyperKitty 1.3.12
idna 3.20
lazr.config 4.0.0
lazr.delegates 4.0.0
mailman 3.3.9
mailman-hyperkitty 1.2.1
mailman-web 0.0.9
mailmanclient 3.3.5
Mako 1.4.3
MarkupSafe 3.0.4
mistune 3.3.4
networkx 3.7
nh3 0.3.7
oauthlib 4.0.0
packaging 26.3
passlib 1.7.4
pip 26.2.1
postorius 1.3.13
psutil 7.2.2
psycopg2-binary 2.9.13
publicsuffix2 2.20191221
pycparser 3.11
Pygments 2.21.0
PyJWT 2.15.1
python-dateutil 2.9.0.post0
python-gettext 5.0
python3-openid 3.2.0
rcssmin 1.3.0
readme_renderer 46.0
requests 2.34.2
rjsmin 1.3.0
robot-detection 0.4
six 1.17.0
SQLAlchemy 2.0.54
sqlparse 0.6.0
typing_extensions 4.16.0
urllib3 2.8.0
uWSGI 2.0.31
wheel 0.48.0
Whoosh 2.7.4
zope.component 7.1
zope.configuration 7.1
zope.event 6.2
zope.hookable 8.3
zope.i18nmessageid 8.3
zope.interface 8.7
zope.schema 8.1
zstd 1.5.7.2
--
hesco(a)yourmessagedelivered.com
912-571-1833 (cell, sms)
3 weeks, 4 days
Re: Mailman 3 deployment: ok to use same database instance for core, web, and hyperkitty tables?
by Andrew Hodgson
Stephen J. Turnbull wrote:
>s.dobrau(a)ucl.ac.uk writes:
> > We are currently working on a migration plan for Mailman -> Mailman > 3. We have around over 3000 mailing lists with a total of 50K > members
>Medium-size. I guess a 2-CPU/4GB Linode or similar should handle the Mailman load (including Postorius and HyperKitty), you might want more memory and maybe more CPUs if you're running the RDBMS on the same host. In the 23k lists/50k users/100k inbound messages/day migration described below, >we ended up reconfiguring the VMs 3 times before declaring the system stable. IMO the 2 x 8CPU/16GB final setup was way overpowered for the need, but we didn't try to calibrate finer than that.
Yep, I have helped a migration with around 150 lists but not as much archived data, and ended up with a 2 CPU with 8GB machine. I think we could have gone lower, but found we needed the larger VM to help us import the archives. I used a managed PostGreSQL instance from Amazon for the database.
> > Our plan is to have core+web (instance 1) and Hyperkitty (instance > 2). With the archives on the FS local to the Hyperkitty.
>The archives are in the database as BLOBs. There's no reason to have a separate instance for HyperKitty. It's not obvious to me that you need more than that 2x4 Linode for both Mailman and RDBMS, but if you're going to split it's Mailman vs RDMBS, not Mailman vs HyperKitty.
I would agree. I think the issue here is there are several ways of splitting out Mailman people get bogged down in whether they should run separate parts on different instances. In my view I have always ran Mailman, Postorius and Hyperkitty on the same host and I think it works better that way. For example, if you run Mailman and the web components on different hosts, you need to make sure the Mailman REST interface is secured, whereas if its on the same host then the REST interface can just listen on localhost. The Dockerised instances do use separate containers for Mailman Core and the web components, but that is the only real time I have seen the components split.
> > And all 3 tables (core, web, Hyperkitty) pointing to the same > database on a remote SQL (mysql-enterprise) instance. Using class > mailman.database.mysql.MySQLDatabase (and whatever necessary on the > Django/Hyperkitty instance).
>Make sure your MySQL database(s) for Mailman are configured with the utf8mb4 option for 4 byte UTF-8 support, or it will choke on emojis and the like. I don't think there are any other gotchas for MySQL.
Are there any performance benchmarks for running MySQL vs PostGreSQL? I tend to fall back to PostGreSQL because it seems to be what is used in a lot of places, I have never recommended going with MySQL for a Mailman instance.
>Note there are a lot more than 3 tables in Mailman's databases. The "traditional" configuration is a "mailman" database for core and a "mailmanweb" database for Django and the archives, but Mark says that as the tables are disjoint across Mailman, Django, and the archives there's no reason not to use a >single "mailman" database for all of them. I believe that's how this list's host is configured.
Yep, the original installation guide created one database for Mailman and the tables for all components are in the same database, this was how my larger install was done. Later revisions of the guide have us split the core vs web components to different databases, I don't think there is any performance hit either way by doing that.
>Importing list configurations, users, and subscriptions into Mailman 3 is pretty fast, as long as you're not using the stock Postfix support.[2] The problem is that generating the Postfix alias files for the lists seems to be noticably linear in number of lists, which means it's quadratic for the mass import. IIRC with >5000 lists you'd be looking at >1m/list just to keep regenerating Postfix's alias database, ie 5000 minutes. I found two solutions, both of which required patching Mailman. I hope to get both into the next release.
>I think we got it down to <5s/list, for 5000 lists that's < 1hr.
That is very interesting I had a similar issue with the Postfix alias generation in my larger setup but didn't have time to identify the root cause. In the end I used Exim sending all mails for the list domain to Mailman which is the setup I use elsewhere and it doesn't rely on the alias file generation.
>Importing archives is ... yeeech. The client decided they didn't want a ton of archives anyway (you know corporations, if it's not required by law, shred it after 6 months). I don't recall the size estimate accurately but we kept 6 months X 4500 lists, maybe 100GB out of 2.3TB of mboxes. That took 24 hours to >import into HyperKitty, without doing the full-text indexing. I do know the original conservative estimate was 20 days to import the whole 2,3 TB. The full-text indexing took more than a week if I remember correctly.
Yes, I had the same problem with the archives. I did import all the archives, in our case it took us several months to get the archives in so as not to stress load on the box. Full text indexing really caused an issue for me, I ended up disabling it for the time we were importing archives then generated a clean index once the archives were imported. This wasn't a good user experience and I would probably plan this out better next time.
>It's also quite possible to migrate incrementally, a few lists at a time. Mailman 2 and Mailman 3 can coexist happily on the same host.
>Mark can advise on that, I think.
Its worth noting this is more difficult to do now on Debian/Ubuntu installs as Python2 has been removed. This was something I ran into myself when trying to co-exist both installs on the same box.
Andrew.
3 years
Re: moving lists to another server
by Mark Sapiro
On 11/21/23 04:32, Odhiambo Washington wrote:
>
> I tested the above out of curiosity. It failed. There are 9 lists in the
> setup, but I am only interested in 1.
>
> ```
> (venv) mailman@debian12:~$ mailman-web shell
> Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
> Type 'copyright', 'credits' or 'license' for more information
> IPython 8.17.2 -- An enhanced Interactive Python. Type '?' for help.
>
> In [1]: from hyperkitty.lib.compat import get_list_by_name
>
> In [2]: mlist = get_list_by_name('list_name_only', 'list_domain')
You can't use literally ('list_name_only', 'list_domain'). If the list
of interest is for example alist(a)example.com, the above should be
mlist = get_list_by_name('alist', 'example.com')
...
> Was I supposed to substitute something for the real listname of the list I
> wanted?
Yes.
> Another thing - totally unrelated - that surprised me:
> While using the commented-out block below, I could not click and get
> anything beyond http://http://list.name/mailman3 as anything to do with
> /postorius/xxxxx gave "Page not found" error.
> However, with the second config block that uses the UNIX socket, I am able
> to access everything
>
> ```
> #<IfModule mod_proxy.c>
> # ProxyPreserveHost On
> # ProxyPass "/mailman3" "http://127.0.0.1:8010/mailman3"
> # ProxyPass "/archives" "http://127.0.0.1:8010/archives"
> # ProxyPass "/accounts" "http://127.0.0.1:8010/accounts"
> # ProxyPass "/admin" "http://127.0.0.1:8010/admin"
> # ProxyPass "/user-profile" "http://127.0.0.1:8010/user-profile"
> # </IfModule>
>
> ProxyPassMatch ^/static/ !
> ProxyPassMatch ^/favicon.ico !
> ProxyPass / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
> ProxyPassReverse / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
> ```
>
> Why is it that the UNIX socket option worked flawlessly while the other
> option failed???
The first option does not have ProxyPass for urls matching /postorius or
/hyperkitty so such urls don't work. The second option passes everything
(ProxyPass / ) to uwsgi via the socket so all urls that uwsgi can handle
will work.
You could do something similar with the first option with simply
ProxyPass "/" "http://127.0.0.1:8010/"
assuming uwsgi is listening on port 8010.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
2 years, 11 months
Re: Opinions wanted.
by John Price
For any setting N items per page, if actualy number of items is less
than or equal to N, no need to render, no need for dropdown. Don't
show it.
If someone is trying to scrap the output for other purposes, perhaps
you can give them an alternative JSON output for that, or something
easler to parse. Or at the very least add an HTML comment with
metadata (like "<!-- total_items=X, page=Y, total_pages=Z -->").
JP
--
How do they get deer to cross the road only at those
yellow road signs?
Have a great day and don't forget to laugh!
http://www.gcfl.net (The Good, Clean Funnies List): Good,
clean daily funnies you can safely tell your Mom!
On Sat, Mar 07, 2026 at 11:51:43AM -0800, Mark Sapiro wrote:
> There is currently a merge request at
> <https://gitlab.com/mailman/django-mailman3/-/merge_requests/345> to hide
> the "Results per page:" dropdown from some Postorius and HyperKitty pages
> when the full list of items (lists) is on the page.
>
> I.e. when the number of items is <=10 and the "Results per page:" is the
> default 10, don't display the dropdown. However there is some controversy
> over whether this should also apply say when the number of items is > 10 but
> still is on one page.
>
> So the question is if we are going to hide the dropdown when "Results per
> page:" is the default 10 and there are <= 10 items so all are on the page,
> should we also hide it when say "Results per page:" is 25 and there are 22
> items all on the page. Your opinion on this is requested via a reply in this
> thread.
>
> See the merge request (link above) for arguments pro and con.
>
> --
> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> San Francisco Bay Area, California better use your sense - B. Dylan
> _______________________________________________
> Mailman-users mailing list -- mailman-users(a)mailman3.org
> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> Archived at: https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message…
>
> This message sent to jp_mailman(a)gcfl.net
8 months
Re: Reject with notification not working
by MegaBrutal
Mark Sapiro <mark(a)msapiro.net> ezt írta (időpont: 2026. márc. 24., K, 8:26):
> On 3/24/26 11:30 AM, MegaBrutal via Mailman-users wrote:
> > I'm creating a newsletter style mailing list where only one address is
> > allowed to send e-mails, members just receive them, but they shouldn't be
> > able to post to the list. Still I want to send a friendly notification if
> > members accidentally try to post to the list, so I set "Reject (with
> > notification)" as "Default action to take when a member posts to the
> list",
> > but it doesn't work: my test address didn't receive any notification
> when I
> > tried to send a message to the list.
>
> What is the moderation action for the address you posted from?
>
I posted if from a member address, and members are set to "Reject (with
notification)". The message didn't get to the list (as expected), but no
notification was sent either – basically the action was like if I set
"Discard (no notification)".
> I tried to define the
> > "list:user:notice:rejected" template for the list, but to no avail.
> What do you mean by no avail? Do you mean the template you defined
> wasn't honored. If so, how did you define it?
>
> Or do you just mean that didn't get you the rejection email?
I mean I didn't get a rejection mail. First I assumed that it's because I
didn't set a rejection message, but setting one did nothing.
Note that I'm managing my list on the Postorius web interface (/mailman3)
and don't really know of any other way of doing it.
Also I created my list as "Announce only", but this didn't seem to have any
effect either. I thought it would create a mailing list that's preset to
behave like a newsletter, but it just created an ordinary list where all
members can post, and I manually needed to set that I don't accept e-mails
from members. I'm puzzled why it even asks for a "List style" in the first
place.
7 months, 2 weeks
how to migrate mailing lists to a new domain using mailman 3
by C. Souza
In a few days my current domain will be disabled (it is out of my control)
and I would like to know how to migrate the mailing lists to the new domain
name.
As a solution I added a new domain in mailman 3 and created a script that
replicates the lists in this new domain. To forward the emails received by
the old lists, the script removes the members and adds the email of the new
list.
Is there a fashionable way to do that?
Also, one problem that I don't have a solution is the archive of the old
lists. How to bring them to the new ones?
Thanks for your help.
My mailing list server is running on a CenOS-7x with postgreSQL and the
following mailman packages:
Package Version
------------------- ----------
1 aiosmtpd 1.2.4
2 alembic 1.5.3
3 arrow 0.17.0
4 asgiref 3.3.1
5 atpublic 2.1.2
6 authheaders 0.13.0
7 authres 1.2.0
8 bleach 3.3.0
9 blessed 1.17.12
10 certifi 2020.12.5
11 cffi 1.14.4
12 chardet 4.0.0
13 click 7.1.2
14 cmarkgfm 0.5.2
15 cryptography 3.3.1
16 decorator 4.4.2
17 defusedxml 0.6.0
18 Django 3.0.12
19 django-allauth 0.44.0
20 django-appconf 1.0.4
21 django-compressor 2.4
22 django-extensions 3.1.0
23 django-gravatar2 1.4.4
24 django-haystack 3.0
25 django-mailman3 1.3.5
26 django-picklefield 3.0.1
27 django-q 1.3.4
28 djangorestframework 3.12.2
29 dkimpy 1.0.5
30 dnspython 2.1.0
31 docutils 0.16
32 falcon 2.0.0
33 flufl.bounce 3.0.1
34 flufl.i18n 2.0.2
35 flufl.lock 3.2
36 gunicorn 20.0.4
37 HyperKitty 1.3.3
38 idna 2.10
39 importlib-resources 5.1.0
40 lazr.config 2.2.3
41 lazr.delegates 2.0.4
42 mailman 3.3.2
43 mailman-hyperkitty 1.1.0
44 mailman-web 0.0.3
45 mailmanclient 3.3.2
46 Mako 1.1.4
47 MarkupSafe 1.1.1
48 networkx 2.5
49 nose 1.3.7
50 oauthlib 3.1.0
51 packaging 20.9
52 passlib 1.7.4
53 pip 21.0.1
54 postorius 1.3.3
55 psycopg2-binary 2.8.6
56 publicsuffix2 2.20191221
57 pycparser 2.20
58 Pygments 2.7.4
59 PyJWT 2.0.1
60 pyparsing 2.4.7
61 python-dateutil 2.8.1
62 python-editor 1.0.4
63 python3-openid 3.2.0
64 pytz 2021.1
65 rcssmin 1.0.6
66 readme-renderer 28.0
67 requests 2.25.1
68 requests-oauthlib 1.3.0
69 rjsmin 1.1.0
70 robot-detection 0.4
71 setuptools 39.2.0
72 six 1.15.0
73 SQLAlchemy 1.3.23
74 sqlparse 0.4.1
75 typing-extensions 3.7.4.3
76 urllib3 1.26.3
77 uWSGI 2.0.19.1
78 wcwidth 0.2.5
79 webencodings 0.5.1
80 wheel 0.36.2
81 Whoosh 2.7.4
82 zipp 3.4.0
83 zope.component 4.6.2
84 zope.configuration 4.4.0
85 zope.deferredimport 4.3.1
86 zope.deprecation 4.4.0
87 zope.event 4.5.0
88 zope.hookable 5.0.1
89 zope.i18nmessageid 5.0.1
90 zope.interface 5.2.0
91 zope.proxy 4.3.5
92 zope.schema 6.0.1
-- Carlos Souza
4 years, 3 months
Re: List creation error
by Brian Carpenter
Odhiambo Washington wrote:
> On Wed, 6 Jan 2021 at 15:36, Andreas Barth aba(a)ayous.org wrote:
> >
> > Odhiambo Washington (odhiambo(a)gmail.com) [210106 12:12]:PS: I could probably have caused the error with some weird operation I
> > did:
> > Initially when postorius comes up, it has the domain - example.com
> > http://example.com. I am not sure
> > where it gets this.
> >
> >
> > example.com is the default domain from django. Renaming that via the
> > django (e.g. mailman3/admin/sites/site/1/change/ ) worked for me quite
> > well (without touching anything in the database by hand).
> > Hi Andi,
> I could be missing something still...
>
> At what juncture during the installation do you do this renaming?
> Where is this directory? On my setup, I have no such directory - but I
> could be missing something in the install process, maybe..
>
> root@debian10:/opt/mailman# pwd
> /opt/mailmanroot@debian10:/opt/mailman# find . -type d -name sites
> ./mm/venv/lib/python3.7/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/sites
> ./mm/venv/lib/python3.7/site-packages/django/contrib/sites
> Do you find anything amiss with this HOWTO -
> https://wiki.list.org/DOC/Howto_Install_Mailman3_On_Debian10 ?? I
> find
> the guide rather straight and clear.
That is my work in progress and thank you for the compliment.
> It's the one I am following - almost to the letter - except for the fact
> that I am using Apache+mod_wsgi+MySQL+Exim instead.
> I have configs in place by the time I run 'mailman info' portion. But then
> again, when I run this command, I don't like what it prints because I see
> some reference to sqlite.
Can you paste what the mailman info output is and the contents of your mailman.cfg file?
> As it is, I believe my problem lies elsewhere, not possible to solve with
> your rename option.
To get rid of example.com, I log into Django, create a new site using the main list domain and note the site id. I then edit the settings_local.py file and change the SITE_ID setting to the new site id of the main list domain. You will need to restart mailman and qcluster I believe after making that change.
5 years, 10 months
Re: Easiest way to install a new mailman3 deployment?
by Andrew Hodgson
Hi,
I completed this on a Debian Buster server instance during last year and have got some notes on the process. I decided to go down the venv route rather than use Docker, but I already had it up and running using the Docker instance on another machine. I found getting it up and running on Docker helped me understand how everything hung together, but ultimately I wanted to run it under a venv so I could understand it a bit more and also I wasn't using Docker in the rest of the environment on that server. I also got caught out earlier on with the Docker containers changing the search engine from Woosh to Xapian which is why I wanted more control over what was going on. Here is a copy of those notes for interest.
I used package managers to provide PostgreSQL support as that was what the Docker install gave me to start with. I may have chosen another DB engine but as I was already using Postgres I stuck with that. I also installed Xapian using the package manager as well as Exim, Nginx, Sassc and Python using the package managers. I didn't install Memcached.
I mainly followed the instructions at: https://wiki.list.org/DOC/Mailman%203%20installation%20experience
My directory structure uses /opt/mailman as the base, the instructions on this web page use /opt/mailman/mm and I didn't have this extra subdirectory. I ensured this directory was owned by list:list.
I installed my venv under /opt/mailman/venv choosing to copy over the existing distribution installed Python packages and installed the following packages via Pip after activating it:
Upgrade packages that are already installed which need an upgrade to work with Mailman3:
pip3 install -U pip zope.interface
Get the rest installed:
pip3 install mailman mailmanclient django-mailman3 mailman-hyperkitty hyperkitty postorius psycopg2-binary xapian-haystack
This also installs Gunicorn which I use as the backend wsgi server.
The reason I chose to use existing system packages as part of the venv is because I wanted to use the Xapian installation I had installed via system packages as I ran into issues installing it via Pip. In future I think I will try and install the PostgreSQL driver this way as well.
Once deactivating the venv I recommend getting the directory structure and files copied to the right places from the links on the wiki, noting that in my cases I had to edit the files to change the relevant paths to suit my new layout, and also where necessary (Systemd units) change the user/group. . You should then run all the Mailman specific commands using these scripts in /opt/mailman/bin. I learnt the hard way that you will run into problems if you try and run Mailman from the venv directory as it will look for files in the wrong place and create data files in the wrong directory. As I went down this road my configuration files mailman.cfg and mailman-hyperkitty.cfg are actually located in /opt/mailman/etc but that isn't a requirement if you follow the wiki and use the scripts provided.
I obtained manage.py, settings.py, urls.py and wsgi.py from the Gitlab Mailman-suite project instead of the wiki site to ensure I got the latest versions and put these files in /opt/mailman. Regarding /opt/mailman/bin/mailman-post/update, I removed the references to Memcached as I didn't use it. I also didn't symlink /opt/mailman/logs to /opt/mailman/var/logs as advised by the wiki but will do this on a reinstall.
I generated a new Mailman PostgreSQL user and restored my existing database to the new server (which also upgraded the DB to a newer Postgresql version as the Docker-Compose from the Mailman Docker project uses an older Postgresql container).
I copied the Mailman runtime files from the Mailman core container data volume to /opt/mailman/var.
I generated a Hyperkitty API secret key and also a Django secret key.
Taking the mailman.cfg file from the wiki as a base, I updated the following parts:
[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://mailman:[password]@localhost/mailmandb
[mailman]
site_owner: mailman-owner(a)lists.domain.com
[mta]
incoming: mailman.mta.exim4.LMTP
configuration: python:mailman.config.exim4
[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /opt/mailman/etc/mailman-hyperkitty.cfg
In /opt/mailman/etc/mailman-hyperkitty.cfg add the following:
[general]
base_url: http://localhost:8000/hyperkitty/
api_key: [generated-hyperkitty-api-key]
Using systemctl see if you can start Mailman and observe logs etc.
In terms of the Django stuff you need to create /opt/mailman/settings-local.py which will be read in by the system and will overwrite the default settings in /opt/mailman/settings.py obtained from the Mailman-suite project. This directly configures the Django framework which the Mailman web components Postorius and Hyperkitty run under.
Here is a outline copy of my file:
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
SECRET_KEY = '[generated-django-secret-key]'
MAILMAN_ARCHIVER_KEY = '[generated-hyperkitty-api-key]'
DEFAULT_FROM_EMAIL = 'mailman-owner(a)lists.domain.com'
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
DEBUG = False
USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
ALLOWED_HOSTS = [
'localhost',
'lists.domain.com'
]
INSTALLED_APPS = (
# Copy the installed apps from the settings.py and remove the social provider logins you don't want to support.
)
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'mailmandb',
'USER': 'mailman',
'PASSWORD': '[password]',
'HOST': 'localhost'
}
}
HAYSTACK_CONNECTIONS = {
'default': {
'PATH': os.path.join(BASE_DIR, "fulltext_index"),
'ENGINE': 'xapian_backend.XapianEngine'
},
}
LOGGING = {
# Copy the complete LOGGING section from settings.py and change the path as follows, if there is a way of just adding the relevant path override to this file let me know. I am doing this only because I didn't create a symlink for /opt/mailman/logs.
'version': 1,
'disable_existing_loggers': False,
'filters': {
[...]
}
},
'handlers': {
'mail_admins': {
[...]
},
'file':{
'level': 'INFO',
'class': 'logging.handlers.WatchedFileHandler',
'filename': os.path.join(BASE_DIR, 'var/logs', 'mailmansuite.log'),
'formatter': 'verbose',
},
'console': {
[...]
},
},
'loggers': {
'django.request': {
[...]
},
'django': {
[...]
},
'hyperkitty': {
[...]
},
'postorius': {
[...]
},
},
'formatters': {
'verbose': {
[...]
},
'simple': {
[...]
},
},
}
Try running /opt/mailman/bin/mailman-post-update and choose to rebuild indexes, see if the static files get saved to /opt/mailman/static and the search indexes get saved to /opt/mailman/fulltext_index. In my case I had plenty of indexing to do, not sure if these files are created on a green-field installation with nothing archived. This command will also populate the Postgresql database schema.
See if the Systemd units for mailman-web and mailman-cluster come up. Observe logs. If everything ok the Gunicorn should be listening on localhost:8000.
For Nginx integration I followed the wiki, only change I had to make was to send the host header to the backend server, here is a partial config file:
server {
server_name lists.domain.com;
root /var/www/lists;
access_log /var/log/nginx/lists-access.log;
error_log /var/log/nginx/lists-error.log warn;
listen [::]:443 ssl; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/lists.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/lists.domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
location = /favicon.ico {
log_not_found off;
}
location = /robots.txt {
log_not_found off;
}
location / {
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host;
proxy_pass http://localhost:8000/;
}
location /static {
alias /opt/mailman/static;
}
}
server {
if ($host = lists.domain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
listen [::]:80;
server_name lists.domain.com;
return 404; # managed by Certbot
}
For Exim integration I followed to the letter the information here: https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#exim
Here is a Logrotate script which is not in the Mailman wiki:
/opt/mailman/var/logs/*.log {
missingok
sharedscripts
su list list
postrotate
/opt/mailman/bin/mailman reopen &>/dev/null || true
if [ -r /opt/mailman/var/gunicorn.pid ]; then
PID=`cat /opt/mailman/var/gunicorn.pid`
kill -s USR1 $PID
fi
endscript
}
I have been running this setup since March 2020, I run around 10 lists on this server with around 10 mails per list per day. I actually had more issues when moving from Mailman2 to Mailman3 using the Docker containers and actually this process made me understand a lot more how everything hangs together and where to troubleshoot issues.
Hope this helps someone out there.
Andrew.
-----Original Message-----
From: Matthew Alberti <matthew(a)alberti.us>
Sent: 01 January 2021 02:48
To: ieso(a)johnwillson.com
Cc: 'Mailman users' <mailman-users(a)mailman3.org>
Subject: [MM3-users] Re: Easiest way to install a new mailman3 deployment?
I had a very similar experience. I don't mean to knock the dev team; Mailman3 is an awesome tool that really meets a need. It is really much much more than a 'tool.' A very comprehensive software suite.
For me, I ended up using the docker option on Ubuntu 20.04. My advice is to give up on the idea of using OS package managers. They just aren't current enough (certainly for Ubuntu LTS), or updated frequently enough, for the current development of Mailman3. Nothing but problems for me on Ubuntu, and Debian is about the same.
Pypi, or installation in a virtualized environment, was equally challenging. I strongly recommend the docker installation method. It is not quite turnkey, but the documentation is close enough where it might 'just work' in your environment. I used MariaDB, and that introduced some challenges. If you are using postgre, postfix, and nginx, you are pretty close to a very easy installation.
This list is very supportive, with very frequent responses from the core dev team. And there are guys like me who really struggled to get to a production instance. Please let us all know what isn't working in the docker approach, and we will try to help I'm sure.
- Matt Alberti
Get BlueMail for Android
On Dec 31, 2020, 8:15 PM, at 8:15 PM, ieso(a)johnwillson.com wrote:
>So I've been trying for the last two weeks to get a new mailman3 server
>running on a virtualized server (any server), and I'm turning to this
>list after having failed many times and running out of holiday time.
>
>I started trying a non-docker installation on Ubuntu 18.04
>(https://docs.google.com/document/d/1xIcSsoNFp2nHi7r4eQys00s9a0k2sHhu1V
>5PlantPTs/edit#) , which got me the closest. Except I had a problem
>with inbound email only being triggered when it came from certain
>accounts. But that clearly wasn't good enough for production, so after
>many attempts to figure out where it was failing, I decided to turn to
>docker as a solution that should be cleaner.
>
>A few attempts at doing a docker installation on digitalocean.com
>failed, which I realized might be due to it not routing private IP
>addresses, so I moved to AWS after checking that their VPC policy would
>fit mailman's docker requirements. I found a great but slightly
>outdated guide on how to do this
>(https://xiaoxing.us/2018/01/01/deploy-mailman-3-on-aws-using-docker/)
>By this point I knew enough to correct a number of places where the
>environment had changed since the procedure was written, but postorius
>still failed at the curl test.
>
>The challenge for me has been the difficulty to know how to
>troubleshoot the different different systems and network infrastructure
>that are used to keep mailman3 humming. I've tried about 7 different
>installation walkthroughs (there are no recent ones on Youtube by the
>way, in case anyone wants to seize that opportunity!), and the good
>guides provide ways to check each stage to try to help you a bit on
>that front.
>
>Nonetheless, I feel stuck and thought I'd ask the simple question...
>for a completely basic, barebones new installation, what's the easiest
>way to get a mailman3 installation up-and running? (e.g. Which server
>provider? Which operating system and version? Docker or otherwise?)
>
>Any pointers highly appreciated. Google Groups is clearly on its way
>out, as it no longer allows for people to easily join groups by sending
>an email or clicking a link, so that should be a big opportunity for
>mailman3 to step up and help give those mailing list migrants a new
>home... which is what we're looking for. We're just not quite as smart
>as you guys. ;-) _______________________________________________
>Mailman-users mailing list -- mailman-users(a)mailman3.org To unsubscribe
>send an email to mailman-users-leave(a)mailman3.org
>https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
_______________________________________________
Mailman-users mailing list -- mailman-users(a)mailman3.org To unsubscribe send an email to mailman-users-leave(a)mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
5 years, 10 months
Re: Mailman3 without Hyperkitty in docker
by Danil Smirnov
Hi Abhilash,
Thank you very much - the previous error is fixed now.
But I have another one now, which causes the same 500 Server Error:
bash-4.3# cat /opt/mailman-web-data/logs/uwsgi-error.log
*** Starting uWSGI 2.0.18 (64bit) on [Fri Mar 1 05:42:09 2019] ***
compiled with version: 6.3.0 on 01 March 2019 01:52:42
os: Linux-3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017
nodename: mailman-web
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /opt/mailman-web
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
setgid() to 101
setuid() to 100
chdir() to /opt/mailman-web
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
building mime-types dictionary from file /etc/mime.types...1168 entry found
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 0.0.0.0:8080 fd 8
uwsgi socket 1 bound to TCP address 0.0.0.0:8000 fd 9
Python version: 3.6.8 (default, Jan 30 2019, 23:58:16) [GCC 6.3.0]
Python main interpreter initialized at 0x55a2df7f57e0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166752 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
Traceback (most recent call last):
File
"/usr/local/lib/python3.6/site-packages/django/utils/module_loading.py",
line 20, in import_string
return getattr(module, class_name)
AttributeError: module 'django.contrib.auth.middleware' has no attribute
'SessionAuthenticationMiddleware'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "wsgi.py", line 38, in <module>
application = get_wsgi_application()
File "/usr/local/lib/python3.6/site-packages/django/core/wsgi.py", line
13, in get_wsgi_application
return WSGIHandler()
File
"/usr/local/lib/python3.6/site-packages/django/core/handlers/wsgi.py", line
136, in __init__
self.load_middleware()
File
"/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line
34, in load_middleware
middleware = import_string(middleware_path)
File
"/usr/local/lib/python3.6/site-packages/django/utils/module_loading.py",
line 24, in import_string
) from err
ImportError: Module "django.contrib.auth.middleware" does not define a
"SessionAuthenticationMiddleware" attribute/class
unable to load app 0 (mountpoint='') (callable not found or import error)
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 37, cores: 2)
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
--- no python application found, check your startup logs for errors ---
As per my investigation, this error possibly caused by Django upgrade:
*The SessionAuthenticationMiddleware class is removed. It provided no
functionality since session authentication is unconditionally enabled in
Django 1.10.*
See https://docs.djangoproject.com/en/2.0/releases/2.0/
After I've removed the line
https://github.com/maxking/docker-mailman/blob/master/postorius/mailman-web…
from settings.py, the container is finally working.
I've placed PR in github:
https://github.com/maxking/docker-mailman/pull/314
Thank you for your help.
Danil
пт, 1 мар. 2019 г. в 04:33, Abhilash Raj <maxking(a)asynchronous.in>:
>
>
> On Thu, Feb 28, 2019, at 5:19 PM, Abhilash Raj wrote:
> > On February 27, 2019 11:53:34 PM PST, Danil Smirnov <danil(a)smirnov.la>
> wrote:
> > >Hi Mark,
> > >
> > >Actually all this stuff happens inside the container after it's started
> > >and
> > >initialized without errors. I didn't amend the image anyhow.
> > >
> > >Also I have the configuration based on docker-compose.yaml provided in
> > >the
> > >repository which works just fine for mailman-web image.
> > >
> > >The very same config with postorius image (+ UWSGI_STATIC_MAP param)
> > >fails
> > >with 500 error...
> > >
> > >Now answering your questions:
> > >
> > >ср, 27 февр. 2019 г., 22:56 Mark Sapiro <mark(a)msapiro.net>:
> > >
> > >> do you have django-q in INSTALLED_APPS in your Django settings and
> > >what
> > >> is set for Q_CLUSTER?
> > >>
> > >
> > >I've checked settings.py inside the container and I haven't found any
> > >of
> > >the mentioned configurations there.
> > >
> > >Does the manage.py that's being run here point to the right
> > >settings.py?
> > >>
> > >
> > >No clue.
> > >
> > >Also I wonder why we have some Hyperkitty-related cronjobs in non-HK
> > >configured image?
> > >
> > >
> https://github.com/maxking/docker-mailman/blob/master/postorius/mailman-web…
> >
> > Because I am a dum dum ;-)
> >
> > Will work on removing those crons and stuff. Would be nice if you
> > opened an issue on Github so that I don't forget.
>
> Should now be fixed in the latest image.
>
> >
> > >Danil
> > >
> > >
> > >
> > >> --
> > >> Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
> > >> San Francisco Bay Area, California better use your sense - B.
> > >Dylan
> > >> _______________________________________________
> > >> Mailman-users mailing list -- mailman-users(a)mailman3.org
> > >> To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> > >> https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> > >>
> > >_______________________________________________
> > >Mailman-users mailing list -- mailman-users(a)mailman3.org
> > >To unsubscribe send an email to mailman-users-leave(a)mailman3.org
> > >https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
> >
> >
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> thanks,
> Abhilash Raj (maxking)
>
7 years, 8 months