Trouble importing a Mailman 3 Postgres database into a new Mailman 3 instance (Docker)
Our in-production Mailman 3 service is running under Docker (maxking's mailman-web, mailman-core and Postgres database containers).
I want to move Mailman to a different server, where I have a clean install of Mailman 3 up and running. When I take a database dump of the in-production database, and import it on the new server, a ton of errors are produced, and the core and web containers die after 15 seconds or so.
I have noticed that if I first delete the mailmandb database on the new server, then create it anew, and _then_ do the import, everything is fine. I see all my lists in the web interface on the new server, it's all good. But deleting and then creating a new blank mailmandb database just prior to importing ... it can't be the right thing to do.
Below is a sample of the errors that are reported on the stdout from my docker-compose command. I'd be very grateful for any pointers as to where to go next with this.
Many thanks in advance, Stephen Kenny
database_1 | ERROR: relation "auth_permission_id_seq" already exists database_1 | STATEMENT: CREATE SEQUENCE public.auth_permission_id_seq database_1 | START WITH 1 database_1 | INCREMENT BY 1 database_1 | NO MINVALUE database_1 | NO MAXVALUE database_1 | CACHE 1;
database_1 | ERROR: insert or update on table "_request"well violates foreign key constraint "_request_mailing_list_id_fkey" database_1 | DETAIL: Key (mailing_list_id)=(774) is not present in table "mailinglist". database_1 | STATEMENT: COPY public._request (id, key, request_type, data_hash, mailing_list_id) FROM stdin;
database_1 | STATEMENT: ALTER TABLE ONLY public.auth_group_permissions database_1 | ADD CONSTRAINT auth_group_permissions_group_id_permission_id_0cd325b0_uniq UNIQUE (group_id, permission_id); database_1 | ERROR: multiple primary keys for table "auth_group_permissions" are not allowed database_1 | STATEMENT: ALTER TABLE ONLY public.auth_group_permissions database_1 | ADD CONSTRAINT auth_group_permissions_pkey PRIMARY KEY (id); database_1 | ERROR: multiple primary keys for table "auth_group" are not allowed database_1 | STATEMENT: ALTER TABLE ONLY public.auth_group database_1 | ADD CONSTRAINT auth_group_pkey PRIMARY KEY (id); database_1 | ERROR: relation "auth_permission_content_type_id_codename_01ab375a_uniq" already exists
database_1 | ERROR: relation "account_emailaddress_user_id_2c513194" already exists database_1 | STATEMENT: CREATE INDEX account_emailaddress_user_id_2c513194 ON public.account_emailaddress USING btree (user_id); database_1 | ERROR: relation "account_emailconfirmation_email_address_id_5b7f8c58" already exists database_1 | STATEMENT: CREATE INDEX account_emailconfirmation_email_address_id_5b7f8c58 ON public.account_emailconfirmation USING btree (email_address_id); database_1 | ERROR: relation "account_emailconfirmation_key_f43612bd_like" already exists database_1 | STATEMENT: CREATE INDEX account_emailconfirmation_key_f43612bd_like ON public.account_emailconfirmation USING btree (key varchar_pattern_ops);
database_1 | ERROR: constraint "hyperkitty_tagging_thread_id_ab30508b_fk_hyperkitty_thread_id" for relation "hyperkitty_tagging" already exists database_1 | STATEMENT: ALTER TABLE ONLY public.hyperkitty_tagging database_1 | ADD CONSTRAINT hyperkitty_tagging_thread_id_ab30508b_fk_hyperkitty_thread_id FOREIGN KEY (thread_id) REFERENCES public.hyperkitty_thread(id) DEFERRABLE INITIALLY DEFERRED; database_1 | ERROR: constraint "hyperkitty_tagging_user_id_9af832e2_fk_auth_user_id" for relation "hyperkitty_tagging" already exists database_1 | STATEMENT: ALTER TABLE ONLY public.hyperkitty_tagging database_1 | ADD CONSTRAINT hyperkitty_tagging_user_id_9af832e2_fk_auth_user_id FOREIGN KEY (user_id) REFERENCES public.auth_user(id) DEFERRABLE INITIALLY DEFERRED;
and finally ....
mailman-core | Traceback (most recent call last): mailman-core | File "/usr/bin/mailman", line 8, in <module> mailman-core | sys.exit(main()) mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ mailman-core | return self.main(*args, **kwargs) mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 1054, in main mailman-core | with self.make_context(prog_name, args, **extra) as ctx: mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 920, in make_context mailman-core | self.parse_args(ctx, args) mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 1613, in parse_args mailman-core | rest = super().parse_args(ctx, args) mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 1378, in parse_args mailman-core | value, args = param.handle_parse_result(ctx, opts, args) mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 2360, in handle_parse_result mailman-core | value = self.process_value(ctx, value) mailman-core | File "/usr/lib/python3.8/site-packages/click/core.py", line 2322, in process_value mailman-core | value = self.callback(ctx, self, value) mailman-core | File "/usr/lib/python3.8/site-packages/mailman/bin/mailman.py", line 95, in initialize_config mailman-core | initialize(value) mailman-core | File "/usr/lib/python3.8/site-packages/mailman/core/initialize.py", line 218, in initialize mailman-core | initialize_2(propagate_logs=propagate_logs) mailman-core | File "/usr/lib/python3.8/site-packages/mailman/core/initialize.py", line 177, in initialize_2 mailman-core | config.db = getUtility(IDatabaseFactory, utility_name).create() mailman-core | File "/usr/lib/python3.8/site-packages/mailman/database/factory.py", line 55, in create mailman-core | SchemaManager(database).setup_database() mailman-core | File "/usr/lib/python3.8/site-packages/mailman/database/factory.py", line 84, in setup_database mailman-core | current_rev = context.get_current_revision() mailman-core | File "/usr/lib/python3.8/site-packages/alembic/runtime/migration.py", line 435, in get_current_revision mailman-core | raise util.CommandError( mailman-core | alembic.util.exc.CommandError: Version table 'alembic_version' has more than one head present; please use get_current_heads() mailman-core exited with code 1 mailman-web | Compiling locale files in /usr/lib/python3.8/site-packages mailman-web | Traceback (most recent call last): mailman-web | File "/opt/mailman-web/manage.py", line 10, in <module> mailman-web | execute_from_command_line(sys.argv) mailman-web | File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line mailman-web | utility.execute() mailman-web | File "/usr/lib/python3.8/site-packages/django/core/management/__init__.py", line 363, in execute mailman-web | settings.INSTALLED_APPS mailman-web | File "/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 82, in __getattr__ mailman-web | self._setup(name) mailman-web | File "/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 69, in _setup mailman-web | self._wrapped = Settings(settings_module) mailman-web | File "/usr/lib/python3.8/site-packages/django/conf/__init__.py", line 170, in __init__ mailman-web | mod = importlib.import_module(self.SETTINGS_MODULE) mailman-web | File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module mailman-web | return _bootstrap._gcd_import(name[level:], package, level) mailman-web | File "<frozen importlib._bootstrap>", line 1014, in _gcd_import mailman-web | File "<frozen importlib._bootstrap>", line 991, in _find_and_load mailman-web | File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked mailman-web | File "<frozen importlib._bootstrap>", line 671, in _load_unlocked mailman-web | File "<frozen importlib._bootstrap_external>", line 848, in exec_module mailman-web | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed mailman-web | File "/opt/mailman-web/settings.py", line 65, in <module> mailman-web | MAILMAN_ARCHIVER_FROM = (os.environ.get('MAILMAN_HOST_IP', gethostbyname(os.environ.get('MAILMAN_HOSTNAME', 'mailman-core'))),) mailman-web | socket.gaierror: [Errno -2] Name does not resolve mailman-web | python3: can't open file 'manage.py': [Errno 2] No such file or directory mailman-web exited with code 2
skenny@scss.tcd.ie writes:
But deleting and then creating a new blank mailmandb database just prior to importing ... it can't be the right thing to do.
If you're using containers, then you're using a system that should be turnkey for the *Mailman* admin, who isn't necessarily the *database* admin. So the database is going to be already populated to some extent. That will cause conflicts when you import a database dump, because the dump expects a blank slate. AFAICS (I haven't done it, so this is just theory) this *is* the right thing to do: blank out the database so the import just works.
Does the installation work when you do it this way? If not, what is the problem?
Hi Stephen,
Yes, everything *does* work if I delete and create the database afresh prior to importing - I see all my lists, with the correct memberships, settings etc.
I was afraid that by removing the default database that comes with the database container, I'd be losing some vital information that may have been added or changed in the database config since I first got my production server working 2 years ago - changed field values, extra table columns ... ? Does that make any sense? I've a hazy understanding at best of the minutiae of databases, being honest.
I had actually started to mull over creating all my lists afresh on the new server, and populating them using scripting/mailmanclient, but I'm now encouraged by your answer :-) - so maybe I'll just wipe and recreate mailmandb, and then do the import.
Thanks a lot for your reply Stephen
On 7/21/22 08:09, skenny@scss.tcd.ie wrote:
I was afraid that by removing the default database that comes with the database container, I'd be losing some vital information that may have been added or changed in the database config since I first got my production server working 2 years ago - changed field values, extra table columns ... ? Does that make any sense? I've a hazy understanding at best of the minutiae of databases, being honest.
You should run your Django admin command with the migrate argument,
however you run that in your container after you import the database to
apply any necessary migrations. This should fix any issues with added
columns or changed definitions.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks Mark, that didn't work out I'm afraid. This is what happened when I tried importing without deleting/recreating mailmandb:
I imported the datatase using a psql command in the database container, and then ran "manage.py migrate", which I think does the same thing as the Django admin command with argument "migrate" (?). It reports "No migrations to apply."
In the web interface the lists appear, but without any members, and as before when I the restart the core and web containers, they die after 15 seconds with the error messages as printed below.
(As I mentioned earlier, removing the newly minted mailmandb database and then creating a new blank one, and importing into that does work well (c.f. Stephen Turnbull's remarks above)).
Stephen
mailman-core | alembic.util.exc.CommandError: Version table 'alembic_version' has more than one head present; please use get_current_heads()
and
mailman-web | MAILMAN_ARCHIVER_FROM = (os.environ.get('MAILMAN_HOST_IP', gethostbyname(os.environ.get('MAILMAN_HOSTNAME', 'mailman-core'))),) mailman-web | socket.gaierror: [Errno -2] Name does not resolve
On 7/22/22 04:52, skenny@scss.tcd.ie wrote:
Thanks Mark, that didn't work out I'm afraid. This is what happened when I tried importing without deleting/recreating mailmandb:
I imported the datatase using a psql command in the database container, and then ran "manage.py migrate", which I think does the same thing as the Django admin command with argument "migrate" (?). It reports "No migrations to apply."
In the web interface the lists appear, but without any members, and as before when I the restart the core and web containers, they die after 15 seconds with the error messages as printed below.
(As I mentioned earlier, removing the newly minted mailmandb database and then creating a new blank one, and importing into that does work well (c.f. Stephen Turnbull's remarks above)).
My point was after you remove the newly minted mailmandb database and
then create a new blank one, and import into that, you should then run
manage.py migrate to apply any new migrations to the imported database.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
My point was after you remove the newly minted mailmandb database and then create a new blank one, and import into that, you should then run manage.py migrate to apply any new migrations to the imported database.
Ah, right ... ok, I've done that now, and all's good.
Actually, I've noticed something related. It looks like the mailman-web container runs manage.py as part of its startup procedure in any event.
After the import completes, I start up the mailman-web container and, its logs say this:
mailman-web | Operations to perform: mailman-web | Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount mailman-web | Running migrations: mailman-web | Applying auth.0012_alter_user_first_name_max_length... OK mailman-web | Applying django_q.0014_schedule_cluster... OK mailman-web | Applying hyperkitty.0022_mailinglist_archive_rendering_mode... OK mailman-web | Applying postorius.0014_auto_20210329_2248... OK mailman-web | Applying postorius.0015_auto_20210619_0509... OK mailman-web | Applying postorius.0016_auto_20210810_2157... OK
If the container is restarted again, it shows that there's nothing to do this time:
mailman-web | Apply all migrations: account, admin, auth, contenttypes, django_mailman3, django_q, hyperkitty, postorius, sessions, sites, socialaccount mailman-web | Running migrations: mailman-web | No migrations to apply.
These latter diagnostics are also what I see if, at this point, I enter the mailman-web container and run "manage.py migrate" explicitly.
I really appreciate the help, thanks so much.
participants (3)
-
Mark Sapiro -
skenny＠scss.tcd.ie -
Stephen J. Turnbull