Re: sqlite3 Database "read only" error on mailman_sync job
by Ugnius
Hi. I have fixed my RO database issue. Thank You for help, Abhilash. It was 100% my stupid fault. I had issues with "django-admin". If I ran manage.py it works correct, if I run django-admin - it not: only basic commands. I did not find way how to give correct environment. Then I found on some notes, that manage.py does exactly the same what django-admin does. So I made work around : I had created /usr/local/bin/django-admin to my project file manage.py and it started to work. Later I installed mailman from 0 under another user, and I forgot about that sym link. So why because database was Read Only. It had simply leaking write privileges to database file. Understandable, because owner was another user...
> On 6 Sep 2019, at 00:20, Abhilash Raj <maxking(a)asynchronous.in> wrote:
>
>> On Thu, Sep 5, 2019, at 1:49 PM, Ugnius wrote:
>>
>> Thank You. This is very valuable information. According documentation I understood that "runserver" should run on some internal port and uwsgi just works like proxy connects to the same as runserver port and serves content to apachi. Because when I was playing with the setup without "runserver" Web interface didn't work.
>> Yes, I have stopped "runserver" and page still works fine. :) This didn't solve me database RO issue, but this information was important. Thank You. I am thinking what to do now... Maybe to recreate database (make initiation once again). It could take 2-3 hours. I did not noticed option to export/import lists setting. Or to try find solution how to make sqlite3 multiple
>> writable.
>
> I think your problem about locked database might be solved. Like I mentioned, it is possible that those two running server could be stepping over each other toes.
>
> After removing the "runserver" process, I think you shouldn't fall into anymore of those locking issues.
>
> FWIW, it is also highly recommended to not use sqlite for any production use cases. Either postgres or mysql is preferred.
>
>> SQLite version 3.16.2 2017-01-06 16:32:41
>> Enter ".help" for usage hints.
>>
>> BR
>> Ugnius
>>
>>> On 5 Sep 2019, at 23:31, Abhilash Raj <maxking(a)asynchronous.in> wrote:
>>>> On Thu, Sep 5, 2019, at 1:15 PM, Ugnius wrote:
>>>>
>>>> I do not know... In the setup example settings was something like process number = 2. But I understand at least 1 uwsgi and 1 run server should run?
>>>
>>> No, only uwsgi needs to run. It is fine if there are two of those, since it probably forks to have 2 worker process. There shouldn't be any "python manage.py runserver" processes running at all, it is meant only for testing purposes.
>>>
>>> The doc mentions that it is "development" server[1][2].
>>>
>>> [1]: # Run the Django's "development" server at localhost:8000
>>> [2]: http://docs.mailman3.org/en/latest/prodsetup.html#setting-up-django-project
>>>
>>> Perhaps it should be more descriptive about what does it mean.
>>>
>>>>
>>>>> On 5 Sep 2019, at 23:08, Abhilash Raj <maxking(a)asynchronous.in> wrote:
>>>>> 6644 ? S 0:12 /usr/bin/uwsgi --master --ini /opt/home/uwsgi.ini
>>>>> 6645 ? S 0:00 /usr/bin/python3 /opt/home/manage.py runserver 0:8015
>>>>> 6653 ? Sl 636:28 /usr/bin/python3 /opt/home/manage.py runserver 0:8015
>>>>> 6656 ? Sl 0:27 /usr/bin/uwsgi --master --ini /opt/home/uwsgi.ini
>>>>>
>>>>> This fishy, why do you have both uwsgi and "python manage.py runserver" running?
>>>>>
>>>>> You only need one of those, since they both do the exact same thing. Please remove
>>>>> whatever is running the "python manage.py runserver" part and only use uwsgi.
>>>>>
>>>>>> On Thu, Sep 5, 2019, at 1:02 PM, Ugnius S wrote:
>>>>>> Thank You, Abhilash, to helping. I have checked the processes, but also I have read, that sqlite3 can run in multi writing mode.
>>>>>> May mailman processes are next:
>>>>>>
>>>>>> 6644 ? S 0:12 /usr/bin/uwsgi --master --ini /opt/home/uwsgi.ini
>>>>>> 6645 ? S 0:00 /usr/bin/python3 /opt/home/manage.py runserver 0:8015
>>>>>> 6653 ? Sl 636:28 /usr/bin/python3 /opt/home/manage.py runserver 0:8015
>>>>>> 6656 ? Sl 0:27 /usr/bin/uwsgi --master --ini /opt/home/uwsgi.ini
>>>>>> 6643 ? S 0:00 /usr/bin/python3 /usr/bin/master -C /opt/home/mailman.cfg
>>>>>> 659 ? S 0:46 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=in:0:1
>>>>>> 6660 ? S 0:42 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=nntp:0:1
>>>>>> 6661 ? S 0:42 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=digest:0:1
>>>>>> 6662 ? S 0:48 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=pipeline:0:1
>>>>>> 6663 ? S 0:44 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=archive:0:1
>>>>>> 6664 ? S 0:41 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=command:0:1
>>>>>> 6665 ? S 0:01 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=retry:0:1
>>>>>> 6666 ? S 0:45 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=virgin:0:1
>>>>>> 6667 ? Sl 0:16 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=lmtp:0:1
>>>>>> 6668 ? Sl 1:30 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=rest:0:1
>>>>>> 6669 ? S 0:42 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=bounces:0:1
>>>>>> 6670 ? S 0:49 /usr/bin/python3 /usr/bin/runner -C /opt/home/mailman.cfg --runner=out:0:1
>>>>>>
>>>>>> Looks like there are no double. Nothing more is running.
>>>>>>
>>>>>> 2019-09-05, kt, 03:16 Abhilash Raj <maxking(a)asynchronous.in> rašė:
>>>>>>
>>>>>>> On Wed, Sep 4, 2019, at 2:05 PM, Ugnius S wrote:
>>>>>>>
>>>>>>> I have installed using documentation. My linux distribution version has no full package available. I have installed step by step:
>>>>>>> $ sudo pip3 install mailman
>>>>>>> $ sudo pip3 install postorius
>>>>>>> $ sudo pip3 install hyperkitty
>>>>>>> $ sudo pip3 install mailman-hyperkitty
>>>>>>> And of course all mandatory components... Everything without errors, I can drop db files and I can download mailman-suite again and start from empty page again. But it will not guarantee that DB will not become READ ONLY again. Here should be logic. No developing support here?
>>>>>>
>>>>>> Could you be running two separate processes such that the database is locked by one and so other finds it as READ Only? When you find a read only database, can you check if it is open in some other process?
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2019-09-03, an, 19:36 Abhilash Raj <maxking(a)asynchronous.in> rašė:
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Sep 1, 2019, at 3:22 AM,ugniusviln(a)gmail.com wrote:
>>>>>>> > Hi. All the project installation is under the same user / path and
>>>>>>> > it works. Archiving working as well, (just one remark: no matter what
>>>>>>> > to set in postorius, hyperkitty shows list as public)
>>>>>>> > I have found work around - it is possible to change list as private in
>>>>>>> > "admin" page.
>>>>>>> > I started to get issues with runjobs after I have created couple of
>>>>>>> > new lists with "no archive" option.
>>>>>>>
>>>>>>> How did you install all these components. It seems there the installation is broken in weird ways.
>>>>>>>
>>>>>>> >
>>>>>>> > If as owner user from project home I run:
>>>>>>> >
>>>>>>> > django-admin mailman_sync
>>>>>>> >
>>>>>>> > Traceback (most recent call last):
>>>>>>> > File
>>>>>>> > "/home/user/.local/lib/python3.5/site-packages/django/db/backends/utils.py",
>>>>>>> > line 84, in _execute
>>>>>>> > return self.cursor.execute(sql, params)
>>>>>>> > File
>>>>>>> > "/home/user/.local/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py",
>>>>>>> > line 383, in execute
>>>>>>> > return Database.Cursor.execute(self, query, params)
>>>>>>> > sqlite3.OperationalError: attempt to write a readonly database
>>>>>>> >
>>>>>>> > "django-admin runjob sync_mailman " - makes read only errors as well.
>>>>>>> > "django-admin runjob new_lists_from_mailman"
>>>>>>> > django.db.utils.OperationalError: attempt to write a readonly database
>>>>>>> > END TRACEBACK
>>>>>>> >
>>>>>>> > I did not find information how to fix this. It was worked without errors
>>>>>>> > before.
>>>>>>> > I noticed, that if I stop mailman, "django-admin mailman_sync" starts
>>>>>>> > to work.
>>>>>>> > If I start "mailman start", then "django-admin mailman_sync" failing
>>>>>>> > again.
>>>>>>> > I did not check other jobs, but I think what is related to database,
>>>>>>> > will get errors.
>>>>>>> > Seems mailman locking the database. It is not a file permissions issue.
>>>>>>> > Mailman db location is: /project_directory/var/data/mailman.db
>>>>>>> >
>>>>>>> > if to rename /project_directory/var/data/mailman.db to something,
>>>>>>> > mailman creates database file size 0, django-admin runjobs start
>>>>>>> > working, but of course postorius stops working. I did it for testing.
>>>>>>> > I have returned this database file back, postorius started to work
>>>>>>> > again, but django-admin runjobs does not work, error: "database read
>>>>>>> > only".
>>>>>>> > Please help me how to fix this issue.
>>>>>>> > _______________________________________________
>>>>>>> > Mailman-users mailing list -- mailman-users(a)mailman3.org
>>>>>>> > To unsubscribe send an email to mailman-users-leave(a)mailman3.org
>>>>>>> > https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
>>>>>>> >
>>>>>>>
>>>>>>> --
>>>>>>> thanks,
>>>>>>> Abhilash Raj (maxking)
>>>>>>
>>>>>> --
>>>>>> thanks,
>>>>>> Abhilash Raj (maxking)
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> thanks,
>>>>> Abhilash Raj (maxking)
>>>>>
>>>>>
>>>
>>> --
>>> thanks,
>>> Abhilash Raj (maxking)
>>>
>>>
>
> --
> thanks,
> Abhilash Raj (maxking)
>
>
5 years, 9 months
Re: moving list from mm3 -> mm3
by Jens Günther
Thanks Mark for our answers, I already got a bit further.
Am 09.12.23 um 00:27 schrieb Mark Sapiro:
>> But it returns {"title": "405 Method Not Allowed"} and I don't know
>> where to look in the docs, since it shows almost always a way through
>> mailman shell.
> You need a PUT command, not POST. Try the above with `-X PUT`.
With the `-X PUT`-command, there is the following error:
{"title": "400 Bad Request", "description": "Unexpected parameters:
bounces_address, created_at, digest_last_sent_at, fqdn_listname,
http_etag, join_address, last_post_at, leave_address, list_name,
mail_host, next_digest_number, no_reply_address, owner_address, post_id,
posting_address, request_address, usenet_watermark, volume"}
-> I don't know why, since I tried it with the exact same example-list
from where I got all these parameters (ex- and then import) ... do I
still miss something? Or does it only work on "fresh" lists?
> You can use Postorius -> Mass operations -> Mass subscribe to add the
> members, but you can't do that for owners and moderators. Also you lose
> information about the members such as Delivery status, Delivery mode and
> other user settings.
Is there another way to preserve these information about the members?
(in general I don't think a lot of people put in effort to customize
there memeberships ... and furthermore they would need to register, right?)
>> ## registered users
>
> That is not the answer. Those are Mailman core users, not Django (web
> login) users. You need to get the relevant entries from the auth_users
> and account_emailaddress tables and add them to the new DB.
Ok, that would be fine.
Are there any more relations to other tables/columns that need to be
set? (I just realised, that there are 2 different id in each table ...
the one in auth_users is the user_id in account_emailaddress :-) scary).
Do you have any suggestions on how to add them correctly?
Thank you in advance for now and another big one in general to all of
you on this list that try to help continouisly with all these problems.
It's a great pleasure to be able to ask for help on a list like that.
Thank you!
Have a good rest of the year. Regards, Jens.
1 year, 5 months
Re: [SOLVED] Re: how to unhold a hold message from a nonmmeber in hold list.
by Odhiambo Washington
On Fri, Apr 21, 2023 at 1:27 PM Guillermo Hernandez (Oldno7) via
Mailman-users <mailman-users(a)mailman3.org> wrote:
>
> El 21/4/23 a las 11:03, Guillermo Hernandez (Oldno7) via Mailman-users
> escribió:
> >
> > El 21/4/23 a las 10:37, Odhiambo Washington escribió:
> >>
> >>
> >> On Fri, Apr 21, 2023 at 11:20 AM Guillermo Hernandez (Oldno7) via
> >> Mailman-users <mailman-users(a)mailman3.org> wrote:
> >>
> >> I'm having troubles with Django and cannot access to web admin.
> >>
> >> I need to unhold a message while figures out what is happening.
> >>
> >> Searching the mailman cli help commands didn't help me.
> >>
> >> Anyone who knows how to unhold a message, please?
> >>
> >>
> >> TBH, I find it a lot easier to fix the web UI than mess with the
> >> python shell :-)
> >
> Well... I found you where right. Messing with the CLI seems a nightmare
> to me.
>
> The problem with the UI seems solved. What I finally have done:
>
> Detected that the module "asgiref" was in a different version that the
> server (with all the same config) that is working. pip upgraded it. This
> didn't work, but it raised new error messages (about hyperkitty parsing
> messages)
>
> Installed pip-review and upgraded ALL python modules with pip-review
> --local --interactive
>
> (Re)upgraded all mailman3 packages with
>
> pip install --upgrade --force-reinstall --no-deps --no-cache-dir
> django-mailman3 mailman mailman-hyperkitty mailmanclient postorius
> django hyperkitty uwsgi mod-wsgi
>
> Restarted apache24
>
> And then all begun to work.
>
> Thanks a lot for your advising and help.
>
Do you now see why a virtualenv style of setup will save you lots of grief?
:-)
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
[How to ask smart questions:
http://www.catb.org/~esr/faqs/smart-questions.html]
2 years, 1 month
Re: MM3 Docker
by Odhiambo Washington
On Sun, 6 Jan 2019 at 23:39, Mark Sapiro <mark(a)msapiro.net> wrote:
> On 1/6/19 8:47 AM, Odhiambo Washington wrote:
> >
> > If you could include the missing steps that you left out then I could
> > possibly be able to follow the steps :-)
>
>
> Please help me understand what steps are missing and I will try to
> provide the information.
Okay.
At the installation stage, you say the following:
"I already had some things set up in /opt/mailman including a git
subdirectory containing clones of the GitLab mailman, mailmanclient,
mailman-hyperkitty, hyperkitty, django-mailman3 and
postorius projects. On the third server I have added mailman-suite
because I use the settings.py from that project as the basis for mine."
Please assume that the reader is a newbie to python3 virtualenv and give the
details of the commands they need to execute to (a) clone all the projects
needed
and (b) create the virtualenv.
The next steps are:
mkdir /opt/mailman/mm
Then you say "I then created a /opt/mailman/mm directory and within that a
Python 3.6 virtualenv"
What happens here is something that needs clarification, because you are
specific about "python3.6".
I have googled and realized that one has to be specific about the python
version they want to use in their virtualenv
as shown here ->
https://stackoverflow.com/questions/45293436/how-to-specify-python-version-…
exact command to execute to create the /opt/mailman/mm/venv
exact command to execute to activate virtualenv
From there, I think the newbie can follow and fumble till they come up with
a working MM3 site.
Come to think of it, your steps kind of look clearer than most that I have
encountered.
I do understand that there are only a few people working on the MM3
development, which I think is
the reason there still isn't a document similar to the ones included with
MM2.1x source under doc/
However, if your steps here are polished a little, they can possibly become
one such doc.
I think MM3 docs would come with three options - virtualenv option, docker
option and package option.
Installing from source seems like something not easily doable.
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)
6 years, 5 months
Re: moving lists to another server
by Mark Sapiro
On 11/21/23 04:32, Odhiambo Washington wrote:
>
> I tested the above out of curiosity. It failed. There are 9 lists in the
> setup, but I am only interested in 1.
>
> ```
> (venv) mailman@debian12:~$ mailman-web shell
> Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0]
> Type 'copyright', 'credits' or 'license' for more information
> IPython 8.17.2 -- An enhanced Interactive Python. Type '?' for help.
>
> In [1]: from hyperkitty.lib.compat import get_list_by_name
>
> In [2]: mlist = get_list_by_name('list_name_only', 'list_domain')
You can't use literally ('list_name_only', 'list_domain'). If the list
of interest is for example alist(a)example.com, the above should be
mlist = get_list_by_name('alist', 'example.com')
...
> Was I supposed to substitute something for the real listname of the list I
> wanted?
Yes.
> Another thing - totally unrelated - that surprised me:
> While using the commented-out block below, I could not click and get
> anything beyond http://http://list.name/mailman3 as anything to do with
> /postorius/xxxxx gave "Page not found" error.
> However, with the second config block that uses the UNIX socket, I am able
> to access everything
>
> ```
> #<IfModule mod_proxy.c>
> # ProxyPreserveHost On
> # ProxyPass "/mailman3" "http://127.0.0.1:8010/mailman3"
> # ProxyPass "/archives" "http://127.0.0.1:8010/archives"
> # ProxyPass "/accounts" "http://127.0.0.1:8010/accounts"
> # ProxyPass "/admin" "http://127.0.0.1:8010/admin"
> # ProxyPass "/user-profile" "http://127.0.0.1:8010/user-profile"
> # </IfModule>
>
> ProxyPassMatch ^/static/ !
> ProxyPassMatch ^/favicon.ico !
> ProxyPass / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
> ProxyPassReverse / unix:/opt/mailman/mm/var/uwsgi.sock|uwsgi://localhost/
> ```
>
> Why is it that the UNIX socket option worked flawlessly while the other
> option failed???
The first option does not have ProxyPass for urls matching /postorius or
/hyperkitty so such urls don't work. The second option passes everything
(ProxyPass / ) to uwsgi via the socket so all urls that uwsgi can handle
will work.
You could do something similar with the first option with simply
ProxyPass "/" "http://127.0.0.1:8010/"
assuming uwsgi is listening on port 8010.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
1 year, 6 months
Re: Mailman 3 deployment: ok to use same database instance for core, web, and hyperkitty tables?
by Andrew Hodgson
Stephen J. Turnbull wrote:
>s.dobrau(a)ucl.ac.uk writes:
> > We are currently working on a migration plan for Mailman -> Mailman > 3. We have around over 3000 mailing lists with a total of 50K > members
>Medium-size. I guess a 2-CPU/4GB Linode or similar should handle the Mailman load (including Postorius and HyperKitty), you might want more memory and maybe more CPUs if you're running the RDBMS on the same host. In the 23k lists/50k users/100k inbound messages/day migration described below, >we ended up reconfiguring the VMs 3 times before declaring the system stable. IMO the 2 x 8CPU/16GB final setup was way overpowered for the need, but we didn't try to calibrate finer than that.
Yep, I have helped a migration with around 150 lists but not as much archived data, and ended up with a 2 CPU with 8GB machine. I think we could have gone lower, but found we needed the larger VM to help us import the archives. I used a managed PostGreSQL instance from Amazon for the database.
> > Our plan is to have core+web (instance 1) and Hyperkitty (instance > 2). With the archives on the FS local to the Hyperkitty.
>The archives are in the database as BLOBs. There's no reason to have a separate instance for HyperKitty. It's not obvious to me that you need more than that 2x4 Linode for both Mailman and RDBMS, but if you're going to split it's Mailman vs RDMBS, not Mailman vs HyperKitty.
I would agree. I think the issue here is there are several ways of splitting out Mailman people get bogged down in whether they should run separate parts on different instances. In my view I have always ran Mailman, Postorius and Hyperkitty on the same host and I think it works better that way. For example, if you run Mailman and the web components on different hosts, you need to make sure the Mailman REST interface is secured, whereas if its on the same host then the REST interface can just listen on localhost. The Dockerised instances do use separate containers for Mailman Core and the web components, but that is the only real time I have seen the components split.
> > And all 3 tables (core, web, Hyperkitty) pointing to the same > database on a remote SQL (mysql-enterprise) instance. Using class > mailman.database.mysql.MySQLDatabase (and whatever necessary on the > Django/Hyperkitty instance).
>Make sure your MySQL database(s) for Mailman are configured with the utf8mb4 option for 4 byte UTF-8 support, or it will choke on emojis and the like. I don't think there are any other gotchas for MySQL.
Are there any performance benchmarks for running MySQL vs PostGreSQL? I tend to fall back to PostGreSQL because it seems to be what is used in a lot of places, I have never recommended going with MySQL for a Mailman instance.
>Note there are a lot more than 3 tables in Mailman's databases. The "traditional" configuration is a "mailman" database for core and a "mailmanweb" database for Django and the archives, but Mark says that as the tables are disjoint across Mailman, Django, and the archives there's no reason not to use a >single "mailman" database for all of them. I believe that's how this list's host is configured.
Yep, the original installation guide created one database for Mailman and the tables for all components are in the same database, this was how my larger install was done. Later revisions of the guide have us split the core vs web components to different databases, I don't think there is any performance hit either way by doing that.
>Importing list configurations, users, and subscriptions into Mailman 3 is pretty fast, as long as you're not using the stock Postfix support.[2] The problem is that generating the Postfix alias files for the lists seems to be noticably linear in number of lists, which means it's quadratic for the mass import. IIRC with >5000 lists you'd be looking at >1m/list just to keep regenerating Postfix's alias database, ie 5000 minutes. I found two solutions, both of which required patching Mailman. I hope to get both into the next release.
>I think we got it down to <5s/list, for 5000 lists that's < 1hr.
That is very interesting I had a similar issue with the Postfix alias generation in my larger setup but didn't have time to identify the root cause. In the end I used Exim sending all mails for the list domain to Mailman which is the setup I use elsewhere and it doesn't rely on the alias file generation.
>Importing archives is ... yeeech. The client decided they didn't want a ton of archives anyway (you know corporations, if it's not required by law, shred it after 6 months). I don't recall the size estimate accurately but we kept 6 months X 4500 lists, maybe 100GB out of 2.3TB of mboxes. That took 24 hours to >import into HyperKitty, without doing the full-text indexing. I do know the original conservative estimate was 20 days to import the whole 2,3 TB. The full-text indexing took more than a week if I remember correctly.
Yes, I had the same problem with the archives. I did import all the archives, in our case it took us several months to get the archives in so as not to stress load on the box. Full text indexing really caused an issue for me, I ended up disabling it for the time we were importing archives then generated a clean index once the archives were imported. This wasn't a good user experience and I would probably plan this out better next time.
>It's also quite possible to migrate incrementally, a few lists at a time. Mailman 2 and Mailman 3 can coexist happily on the same host.
>Mark can advise on that, I think.
Its worth noting this is more difficult to do now on Debian/Ubuntu installs as Python2 has been removed. This was something I ran into myself when trying to co-exist both installs on the same box.
Andrew.
1 year, 7 months
how to migrate mailing lists to a new domain using mailman 3
by C. Souza
In a few days my current domain will be disabled (it is out of my control)
and I would like to know how to migrate the mailing lists to the new domain
name.
As a solution I added a new domain in mailman 3 and created a script that
replicates the lists in this new domain. To forward the emails received by
the old lists, the script removes the members and adds the email of the new
list.
Is there a fashionable way to do that?
Also, one problem that I don't have a solution is the archive of the old
lists. How to bring them to the new ones?
Thanks for your help.
My mailing list server is running on a CenOS-7x with postgreSQL and the
following mailman packages:
Package Version
------------------- ----------
1 aiosmtpd 1.2.4
2 alembic 1.5.3
3 arrow 0.17.0
4 asgiref 3.3.1
5 atpublic 2.1.2
6 authheaders 0.13.0
7 authres 1.2.0
8 bleach 3.3.0
9 blessed 1.17.12
10 certifi 2020.12.5
11 cffi 1.14.4
12 chardet 4.0.0
13 click 7.1.2
14 cmarkgfm 0.5.2
15 cryptography 3.3.1
16 decorator 4.4.2
17 defusedxml 0.6.0
18 Django 3.0.12
19 django-allauth 0.44.0
20 django-appconf 1.0.4
21 django-compressor 2.4
22 django-extensions 3.1.0
23 django-gravatar2 1.4.4
24 django-haystack 3.0
25 django-mailman3 1.3.5
26 django-picklefield 3.0.1
27 django-q 1.3.4
28 djangorestframework 3.12.2
29 dkimpy 1.0.5
30 dnspython 2.1.0
31 docutils 0.16
32 falcon 2.0.0
33 flufl.bounce 3.0.1
34 flufl.i18n 2.0.2
35 flufl.lock 3.2
36 gunicorn 20.0.4
37 HyperKitty 1.3.3
38 idna 2.10
39 importlib-resources 5.1.0
40 lazr.config 2.2.3
41 lazr.delegates 2.0.4
42 mailman 3.3.2
43 mailman-hyperkitty 1.1.0
44 mailman-web 0.0.3
45 mailmanclient 3.3.2
46 Mako 1.1.4
47 MarkupSafe 1.1.1
48 networkx 2.5
49 nose 1.3.7
50 oauthlib 3.1.0
51 packaging 20.9
52 passlib 1.7.4
53 pip 21.0.1
54 postorius 1.3.3
55 psycopg2-binary 2.8.6
56 publicsuffix2 2.20191221
57 pycparser 2.20
58 Pygments 2.7.4
59 PyJWT 2.0.1
60 pyparsing 2.4.7
61 python-dateutil 2.8.1
62 python-editor 1.0.4
63 python3-openid 3.2.0
64 pytz 2021.1
65 rcssmin 1.0.6
66 readme-renderer 28.0
67 requests 2.25.1
68 requests-oauthlib 1.3.0
69 rjsmin 1.1.0
70 robot-detection 0.4
71 setuptools 39.2.0
72 six 1.15.0
73 SQLAlchemy 1.3.23
74 sqlparse 0.4.1
75 typing-extensions 3.7.4.3
76 urllib3 1.26.3
77 uWSGI 2.0.19.1
78 wcwidth 0.2.5
79 webencodings 0.5.1
80 wheel 0.36.2
81 Whoosh 2.7.4
82 zipp 3.4.0
83 zope.component 4.6.2
84 zope.configuration 4.4.0
85 zope.deferredimport 4.3.1
86 zope.deprecation 4.4.0
87 zope.event 4.5.0
88 zope.hookable 5.0.1
89 zope.i18nmessageid 5.0.1
90 zope.interface 5.2.0
91 zope.proxy 4.3.5
92 zope.schema 6.0.1
-- Carlos Souza
2 years, 11 months
Re: List creation error
by Brian Carpenter
Odhiambo Washington wrote:
> On Wed, 6 Jan 2021 at 15:36, Andreas Barth aba(a)ayous.org wrote:
> >
> > Odhiambo Washington (odhiambo(a)gmail.com) [210106 12:12]:PS: I could probably have caused the error with some weird operation I
> > did:
> > Initially when postorius comes up, it has the domain - example.com
> > http://example.com. I am not sure
> > where it gets this.
> >
> >
> > example.com is the default domain from django. Renaming that via the
> > django (e.g. mailman3/admin/sites/site/1/change/ ) worked for me quite
> > well (without touching anything in the database by hand).
> > Hi Andi,
> I could be missing something still...
>
> At what juncture during the installation do you do this renaming?
> Where is this directory? On my setup, I have no such directory - but I
> could be missing something in the install process, maybe..
>
> root@debian10:/opt/mailman# pwd
> /opt/mailmanroot@debian10:/opt/mailman# find . -type d -name sites
> ./mm/venv/lib/python3.7/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/sites
> ./mm/venv/lib/python3.7/site-packages/django/contrib/sites
> Do you find anything amiss with this HOWTO -
> https://wiki.list.org/DOC/Howto_Install_Mailman3_On_Debian10 ?? I
> find
> the guide rather straight and clear.
That is my work in progress and thank you for the compliment.
> It's the one I am following - almost to the letter - except for the fact
> that I am using Apache+mod_wsgi+MySQL+Exim instead.
> I have configs in place by the time I run 'mailman info' portion. But then
> again, when I run this command, I don't like what it prints because I see
> some reference to sqlite.
Can you paste what the mailman info output is and the contents of your mailman.cfg file?
> As it is, I believe my problem lies elsewhere, not possible to solve with
> your rename option.
To get rid of example.com, I log into Django, create a new site using the main list domain and note the site id. I then edit the settings_local.py file and change the SITE_ID setting to the new site id of the main list domain. You will need to restart mailman and qcluster I believe after making that change.
4 years, 5 months
Re: Digest Formatting
by Mark Sapiro
On 11/7/20 9:08 AM, Stephen J. Turnbull wrote:
>
> I thought there was a way to trigger immediate distribution of a
> digest immediately, but I guess I'm remembering that from Mailman 2.
> Neither the Postorius management interface nor the core documentation
> suggests there's such a trigger in Mailman 3.
There is nothing in the web UI or REST, but the `mailman digests`
command can send accumulated digests for one or more named lists or all
lists unconditionally or only if the list's digest_send_periodic is set
to True. see `mailkman digests --help` for details.
> > 2) I have customized list:member:digest:header, but am not seeing
> > the custom text where I thought it should be. The documentation
> > seems to imply that both list:member:digest:header and
> > list:member:digest:footer surround each message in the digest
>
> Which documentation do you mean? As far as I know (Mark will have
> authoritative information if I'm wrong), the digest header and footer
> appear at the top and bottom of the *whole* digest message (ie, once
> each per digest), not surrounding or in the bodies of the component
> messages.
Steve is correct. The layout is as follows
The masthead from the list:member:digest:masthead template.
The digest header if any from the list:member:digest:header template.
The table of contents.
The individual messages.
The digest footer from the list:member:digest:footer template.
For the MIME digest, these are all separate MIME parts and the
individual messages are message/rfc822 parts within a multipart/digest part.
For the plain text digest, the digest is one text/plain part containing
the above formatted par RFC 1153
And in another post, mailman(a)manygoodideas.com wrote:
> I also have a feature request (I didn't see it in the docs), consider creating list:member:digest:mastfoot – a chunk of text that could be at the end of the digest. Thanks.
This already exists, it is list:member:digest:footer. Your confusion
seems to be because you think the list:member:digest:header template and
list:member:digest:footer template are prepended/appended to each
individual message in the digest which would not make sense and is not
the case.
--
Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers,
San Francisco Bay Area, California better use your sense - B. Dylan
4 years, 7 months
Re: Apache+mod_wsgi issue
by Eggert Ehmke
Am Montag, 26. Dezember 2022, 16:08:42 CET schrieb Odhiambo Washington:
> On Mon, Dec 26, 2022 at 5:07 PM Eggert Ehmke <eggert(a)eehmke.de> wrote:
> > Ok,
> >
> > I hope this is readable. I just copíed the content of some files into the
> > pastebin, don't know if it is possible to post tar.gz there.
> > https://pastebin.ubuntu.com/p/mmRJJwKDc3/
> >
> > Just tell me if this works for you, and when something is missing.
> > Most of the settings are copied from some other sites and just adjusted
> > for my
> > needs.
> > Good luck!
>
> So it seems your WSGIDaemonProcess is defined somewhere outside the VHOST,
> right? Or it wasn't necessary?
I have no settings for WSGIDaemon whatsoever. It's all handled by the uwsgi
inside the venv.
> And what does your urls.py look like?
Which of the various urls.py files do you need?
>
> My setup is PUBLIC for everyone so I will not hide the domain name :-)
>
> In my setup, MM3 is installed in /opt/mailman/mm/. and mod_wsgi is enabled
> in Apache.
> I am running uwsgi on a TCP socket, not unix socket.
>
> The whole of my *VHOST:*
> <CUT>
> WSGIDaemonProcess mailman-web display-name=mailman-web
> maximum-requests=1000 umask=0002 user=mailman group=mailman
> python-path=/opt/mailman/mm:/opt/mailman/mm/venv/lib/python3.9/site-packages
> :/opt/mailman/mm/venv/lib/python3.9 python-home=/opt/mailman/mm/venv
> home=/opt/mailman/mm/var
> WSGIRestrictSignal Off
> <VirtualHost *:443>
> ServerName mm3-lists.kictanet.or.ke
>
> SSLEngine on
> SSLCertificateFile "/usr/local/etc/letsencrypt/live/
> mm3-lists.kictanet.or.ke/cert.pem"
> SSLCertificateKeyFile "/usr/local/etc/letsencrypt/live/
> mm3-lists.kictanet.or.ke/privkey.pem"
> SSLCertificateChainFile "/usr/local/etc/letsencrypt/live/
> mm3-lists.kictanet.or.ke/chain.pem"
> CustomLog /var/log/mm3-lists-access.log combined
> ErrorLog /var/log/mm3-lists-error.log
> LogLevel info
> Alias /favicon.ico /opt/mailman/mm/static/hyperkitty/img/favicon.ico
> Alias /static "/opt/mailman/mm/static"
> <Directory "/opt/mailman/mm/static">
> Require all granted
> </Directory>
> WSGIScriptAlias / /opt/mailman/mm/wsgi.py
> <Directory "/opt/mailman/mm/">
> <Files wsgi.py>
> Order deny,allow
> Allow from all
> Require all granted
> </Files>
> WSGIProcessGroup mailman-web
> </Directory>
> </VirtualHost>
> </CUT>
>
> *2. uwsgi.ini:*
> <CUT>
> [uwsgi]
> http-socket = 127.0.0.1:8000
> pidfile = /opt/mailman/mm/var/uwsgi.pid
> chdir = /opt/mailman/mm/
> virtualenv = /opt/mailman/mm/venv/
> module=mailman_web.wsgi:application
> env =PYTHONPATH=/opt/mailman/mm/
> env =DJANGO_SETTINGS_MODULE=settings
> master = true
> process = 2
> threads = 2
> attach-daemon = /opt/mailman/mm/venv/bin/mailman-web qcluster
> req-logger = file:/opt/mailman/mm/var/logs/uwsgi.log
> logger = qcluster file:/opt/mailman/mm/var/logs/uwsgi-qcluster.log
> log-route = qcluster uwsgi-daemons
> logger = cron file://opt/mailman/mm/var/logs/uwsgi-cron.log
> log-route = cron uwsgi-cron
> logger = file:/opt/mailman/mm/var/logs/uwsgi-error.log
> uid = mailman
> gid = mailman
> </CUT>
>
> *3. uwsgi - *Running supervised (supervisord):
> <CUT>
> [program:uwsgi]
> directory=/opt/mailman/mm/
> environment=PYTHONPATH=/opt/mailman/mm/
> command=/usr/bin/su -m mailman -c '/opt/mailman/mm/venv/bin/uwsgi --ini
> /opt/mailman/mm/uwsgi.ini' 2>&1 | logger -t uwsgi
> autostart=true
> autorestart=true
> user=mailman
> group=mailman
> stopasgroup=true
> killasgroup=true
> </CUT>
>
> *4. urls.py*:
> <CUT>
> from django.conf.urls import include
> from django.urls import re_path
> from django.contrib import admin
> from django.urls import reverse_lazy
> from django.views.generic import RedirectView
> from django_mailman3.views.profile import delete_account, user_profile
> urlpatterns = [
> re_path(r'^$', RedirectView.as_view(
> url=reverse_lazy('list_index'),
> permanent=True)),
> re_path(r'^postorius/', include('postorius.urls')),
> re_path(r'^hyperkitty/', include('hyperkitty.urls')),
> re_path(r'^user-profile/delete$', delete_account,
> name='mm_user_account_delete'),
> re_path(r'^user-profile/$', user_profile, name='mm_user_profile'),
> re_path(r'^accounts/', include('allauth.urls')),
> re_path(r'^admin/', admin.site.urls),
> re_path(r'^mm/', include('postorius.urls')),
> re_path(r'^archives/', include('hyperkitty.urls')),
> ]
> </CUT>
>
> And that's pretty much what I am using on my FreeBSD server.
>
>
> PS: Possible to bottom-post instead of top-posting? :)
2 years, 5 months