In the OpenDev Collaboratory we've got a bunch of separate Mailman 2.1 vhosts we've started migrating into a single Mailman 3.3.7 server, desiring to take full advantage of its improved multi-domain support. For the most part everything is working great (kudos to everyone who maintains the software and documentation, it's excellent).
One of the things our users noticed fairly quickly was Hyperkitty on the second vhost was incorrectly showing the FQDN of the first vhost near the top of its pages. This is covered in the FAQ of course, and seems to imply that what we want is to switch the SITE_ID from 1 to 0 in our settings.py in order to take advantage of magic domain-guessing behavior. So far so good.
Unfortunately, SITE_ID = 0 seems to explicitly require the addition of "mail hosts" (or "mail domains" from Django's perspective), and all the web routing returns error pages for any list sites besides the first one since, by default, our server did not have these predefined. What's odd is that, if I log into Postorius as admin and visit the "manage domains" page, suddenly the database gets populated with mail hosts for all of the FQDNs associated with our imported mailing lists. I would not have expected merely visiting that page as an admin to have a side effect of adding entries for the "missing" mail hosts, is that an intentional behavior or merely a bug?
Once these entries are automatically added by visiting that page, they're all associated with the one "web host" ("site" in Django) we have for the server, and now the list filtering functionality in Postorius breaks and any of the vhosts all display the full set of mailing lists on the server rather than only those associated with a particular vhost (note that this seems to only be confusing Postorius, Hyperkitty has them correctly filtered still).
Okay, so it looks like this is because the mail hosts are all tied to the one web host. We can create additional "sites" in the Django admin UI and then switch the association of the mail hosts to those distinct web hosts. But after doing so, the first vhost still seems to show an unfiltered list of all lists in Postorius (Hyperkitty continues to be fine), while Postorius on the other vhosts correctly shows their filtered sets of lists instead.
I also noticed when mailman-web is starting up, its log claims the first site is being set as the "default domain," so have to wonder if that's related to this behavior. Should we create an extra web host to serve as the "default" so that all our typical vhosts get the expected list filtering behavior, or am I completely off-base and there's a much better approach I've missed?
Sorry this is so long, I feel like I have at most a tenuous grasp of the details and variables involved, but hopefully it makes sense to someone. Thanks in advance for any recommendations or guidance.
Jeremy Stanley
On 12/14/22 13:40, Jeremy Stanley wrote:
Once these entries are automatically added by visiting that page, they're all associated with the one "web host" ("site" in Django) we have for the server, and now the list filtering functionality in Postorius breaks and any of the vhosts all display the full set of mailing lists on the server rather than only those associated with a particular vhost (note that this seems to only be confusing Postorius, Hyperkitty has them correctly filtered still).
See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... and also below.
Okay, so it looks like this is because the mail hosts are all tied to the one web host.
Yes.
We can create additional "sites" in the Django admin UI and then switch the association of the mail hosts to those distinct web hosts. But after doing so, the first vhost still seems to show an unfiltered list of all lists in Postorius (Hyperkitty continues to be fine), while Postorius on the other vhosts correctly shows their filtered sets of lists instead.
I also noticed when mailman-web is starting up, its log claims the first site is being set as the "default domain," so have to wonder if that's related to this behavior. Should we create an extra web host to serve as the "default" so that all our typical vhosts get the expected list filtering behavior, or am I completely off-base and there's a much better approach I've missed?
I'm not really sure what's going on here. You could try creating a dummy default host to see if that fixes the issue for you.
There are subtle differences in the implementation between HyperKitty and Postorius that allow HyperKitty to work better, but it too will exhibit some anomalous behaviors.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2022-12-14 17:26:59 -0800 (-0800), Mark Sapiro wrote:
On 12/14/22 13:40, Jeremy Stanley wrote:
Once these entries are automatically added by visiting that page, they're all associated with the one "web host" ("site" in Django) we have for the server, and now the list filtering functionality in Postorius breaks and any of the vhosts all display the full set of mailing lists on the server rather than only those associated with a particular vhost (note that this seems to only be confusing Postorius, Hyperkitty has them correctly filtered still).
See https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/... and also below. [...]
While this does appear to solve the problem on my test deployments, a challenge that I'm left with is how to automate it. Our community takes an "infrastructure as code" approach to all the services we run for our projects, and we make sure that we can automatically bootstrap every system from scratch into a working state. Unfortunately, I'm not having any luck finding an API or CLI which I can use to create sites and domains rather than manually doing it in the Django admin WebUI. Does such a thing exist, or an I going to need to resort to directly performing database inserts (which I'd rather avoid since it could break if the schema changes). It appears there's a foreign key relationship involved, which could make DB manipulation like this even more fragile.
Jeremy Stanley
On 1/30/23 10:32, Jeremy Stanley wrote:
I'm not having any luck finding an API or CLI which I can use to create sites and domains rather than manually doing it in the Django admin WebUI. Does such a thing exist, or an I going to need to resort to directly performing database inserts (which I'd rather avoid since it could break if the schema changes). It appears there's a foreign key relationship involved, which could make DB manipulation like this even more fragile.
Domains can be created via Postorius or via the Mailman Core REST API
that postorius uses or implicitly by the mailman create command when
creating a list in a new domain. However, none of this can create a new
site. You can do this through the Django admin site at sites/site/add/.
See https://docs.djangoproject.com/en/4.1/ref/contrib/sites/ for methods you can use to do this programmatically.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2023-01-30 12:08:10 -0800 (-0800), Mark Sapiro wrote: [...]
Domains can be created via Postorius or via the Mailman Core REST API that postorius uses or implicitly by the
mailman createcommand when creating a list in a new domain. However, none of this can create a new site. You can do this through the Django admin site at sites/site/add/.See https://docs.djangoproject.com/en/4.1/ref/contrib/sites/ for methods you can use to do this programmatically.
Thanks, I'm still doing my best to wrap my brain around how Django uses database migrations to create sites (seems like a very strange use of a migration, I'm only accustomed to seeing them applied for database schema updates). Once I work out how to script site creation, I'm still not completely finding the explanations I need for how to map them to domains in Mailman though.
The Mailman documentation about domains says "Domains are how Mailman interacts with email host names and web host names" but doesn't actually go on to say (that I can find anywhere) how it actually associates them. I can see in the API docs that a POST to the domains endpoint will create a new domain with a specific mail_host FQDN, but there's some association somewhere of that to a web host (Django site), right? How is that set or manipulated through the API? Or is that something which has to be set directly in Django's data structures instead?
To recap, on creation of a mailing list server I want to populate it automatically with multiple lists on different domains and have dynamic SITE_ID=0 domain guessing performed so that the correct site name and list filtering is performed depending on which domain name is used in a URL the client requests. I believe I need to have the automation perform the following steps:
Instantiate the Mailman installation and run initial migrations with SITE_ID=1 in order to avoid the problem mentioned in the docs about not using SITE_ID=0 until the tables have been populated, then switch the settings.py to use SITE_ID=0 once done.
Create the individual sites I need in Django (Mailman web hosts) by creating and applying database migrations.
Create the individual domains I need in Mailman (Mailman mail hosts) through its REST API.
"Somehow" tell Mailman (or Django?) which mail hosts correspond to what web hosts.
As I said, I'm still trying to understand how to make step 2 happen (Django has a steep learning curve for me), but beyond that I'm unsure how to do step 4 at all. I can do all of this through the WebUI of course, and have confirmed that it works as expected when I do, I'm just trying to work out the automation so that our other sysadmins don't have to perform these steps by hand the next time we need to rebuild the server.
Thanks again for all the help so far, and for any additional insight anyone may have into how I can correct my mental model of all this.
Jeremy Stanley
On 2/15/23 07:52, Jeremy Stanley wrote:
Thanks, I'm still doing my best to wrap my brain around how Django uses database migrations to create sites (seems like a very strange use of a migration, I'm only accustomed to seeing them applied for database schema updates).
I don't think it does. There are a couple of migrations at django/contrib/sites/migrations, but these are only the initial creation of the model and an update to make the domain field unique.
Why do you think it creates sites in this way.
Once I work out how to script site creation, I'm still not completely finding the explanations I need for how to map them to domains in Mailman though.
The Mailman documentation about domains says "Domains are how Mailman interacts with email host names and web host names" but doesn't actually go on to say (that I can find anywhere) how it actually associates them.
Mailman core doesn't associate domains with Django sites. Mailman core knows nothing about a web UI. The association between Mailman core domains and Django sites is via Postorius. It doesn't exist in Mailman core. When Postorius sees a new domain from core, it assigns it a web_host based on the current web host accessing Postorius.
I can see in the API docs that a POST to the domains endpoint will create a new domain with a specific mail_host FQDN, but there's some association somewhere of that to a web host (Django site), right?
No. The only association is if Postorius is creating the domain in core, vs something else creating it via the REST API, Postorius will assign it a web_host in it's own Domain model, and that web host must be an existing Django site, which is only created via Django.
How is that set or manipulated through the API? Or is that something which has to be set directly in Django's data structures instead?
Yes, A new site has to be created in Django. Even Postorius doesn't do that.
To recap, on creation of a mailing list server I want to populate it automatically with multiple lists on different domains and have dynamic SITE_ID=0 domain guessing performed so that the correct site name and list filtering is performed depending on which domain name is used in a URL the client requests. I believe I need to have the automation perform the following steps:
Instantiate the Mailman installation and run initial migrations with SITE_ID=1 in order to avoid the problem mentioned in the docs about not using SITE_ID=0 until the tables have been populated, then switch the settings.py to use SITE_ID=0 once done.
Create the individual sites I need in Django (Mailman web hosts) by creating and applying database migrations.
No. You create the sites via Django, but not by creating and applying database migrations.
Create the individual domains I need in Mailman (Mailman mail hosts) through its REST API.
"Somehow" tell Mailman (or Django?) which mail hosts correspond to what web hosts.
Neither Mailman nor Django. It's Postorius which maintains the relationship between Django sites which it calls web_hosts and domains.
As I said, I'm still trying to understand how to make step 2 happen (Django has a steep learning curve for me), but beyond that I'm unsure how to do step 4 at all. I can do all of this through the WebUI of course, and have confirmed that it works as expected when I do, I'm just trying to work out the automation so that our other sysadmins don't have to perform these steps by hand the next time we need to rebuild the server.
Step 2.
$ django_admin shell
>>> from django.contrib.sites.models import Site
>>> new_site = Site(id=5, domain='5.example.com', name='display_name')
>>> new_site.save()
Of course the values are up to you and you can omit the id= to have Django assign the first available..
Once you've done step 2, you need to create a domain in Postorius. Look at the code behind Postorius add new domain. Or you can create the domain in core via REST, but you then have to modify it in Postorius to associate the correct site.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2/15/23 18:35, Mark Sapiro wrote:
$ django_admin shell
from django.contrib.sites.models import Site new_site = Site(id=5, domain='5.example.com', name='display_name') new_site.save()
Of course the values are up to you and you can omit the id= to have Django assign the first available..
Also note that the above is a django_admin shell interaction. You can also do this in python, but you need to precede the above with
>>> import os
>>> import sys
>>> sys.path.insert(0, '/path/to/directory/containing/settings.py')
>>> os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
>>> import django
>>> django.setup()
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2023-02-15 18:35:35 -0800 (-0800), Mark Sapiro wrote:
On 2/15/23 07:52, Jeremy Stanley wrote:
Thanks, I'm still doing my best to wrap my brain around how Django uses database migrations to create sites (seems like a very strange use of a migration, I'm only accustomed to seeing them applied for database schema updates).
I don't think it does. There are a couple of migrations at django/contrib/sites/migrations, but these are only the initial creation of the model and an update to make the domain field unique.
Why do you think it creates sites in this way. [...]
Clearly a total misreading on my part of the Django documentation to which you referred earlier. It talks about having to feed Django multiple settings.py files with different SITE_ID values and running migrate to configure them, then links to the document on migrations which starts out explaining how to use makemigrations to create them. I took that to mean creating and configuring Django sites from the command line required supplying customized database migrations.
The particularly misleading bits are where it says:
"To set the correct name and domain for your project, you can
use a data migration."
"In order to serve different sites in production, you’d create a
separate settings file with each SITE_ID"It doesn't seem to talk about using django_admin at all (or if it does, it's not mentioned by name, maybe assuming the reader knows to use it in order to perform the necessary steps).
Thanks for the clear examples! If these are available somewhere in Django's documentation, I did a terrible job of searching for them.
you need to create a domain in Postorius. Look at the code behind Postorius add new domain. Or you can create the domain in core via REST, but you then have to modify it in Postorius to associate the correct site. [...]
Much appreciated, this was the other bit that was non-obvious to me. I guess you're saying Postorius's view of domains is independent from what's in Mailman Core, and there's no documented CLI/API for manipulating that in Postorius so I'll need to write one based on a reverse-engineering of its source code? Sounds doable now that I know where to look. Thanks again!
Jeremy Stanley
On 2/16/23 05:34, Jeremy Stanley wrote:
Clearly a total misreading on my part of the Django documentation to which you referred earlier. It talks about having to feed Django multiple settings.py files with different SITE_ID values and running migrate to configure them, then links to the document on migrations which starts out explaining how to use makemigrations to create them. I took that to mean creating and configuring Django sites from the command line required supplying customized database migrations.
The particularly misleading bits are where it says:
"To set the correct name and domain for your project, you can use a data migration." "In order to serve different sites in production, you’d create a separate settings file with each SITE_ID"It doesn't seem to talk about using django_admin at all (or if it does, it's not mentioned by name, maybe assuming the reader knows to use it in order to perform the necessary steps).
Yes, that doc really isn't appropriate for your purpose, but it's the only one I could find on Django sites. Most of what it talks about is for more complex situations than ours and whatever bits are relevant are already done in Postorius/HyperKitty. Sorry for leading you astray.
Thanks for the clear examples! If these are available somewhere in Django's documentation, I did a terrible job of searching for them.
I don't think Django's docs address the simple task of adding a site.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 2023-02-16 11:06:40 -0800 (-0800), Mark Sapiro wrote: [...]
Yes, that doc really isn't appropriate for your purpose, but it's the only one I could find on Django sites. Most of what it talks about is for more complex situations than ours and whatever bits are relevant are already done in Postorius/HyperKitty. Sorry for leading you astray. [...]
Please don't apologize, you've gone well out of your way to be helpful and it's very much appreciated. I've been on the other side of this conversation for plenty of projects and understand the challenges.
I don't think Django's docs address the simple task of adding a site.
While part of me wants to find that hilarious, I can't honestly claim the projects I work on are any better in that regard. Your latest recommendations have me un-stuck and left with a far better sense of how this fits together now.
Jeremy Stanley
participants (2)
-
Jeremy Stanley -
Mark Sapiro