Mailman recovery test did not go as planned
I have a test Mailman 3 environment. We are using docker, and we are running the database outside of docker in mysql. I need to perform a DR recovery test.
I shut down Mailman in docker (mailman-core & mailman-web, the only docker images I have running for Mailman). While Mailman was stopped, I had our database team take a backup dump of the database. I then restarted Mailman in docker. I threw some more messages at a test list. I validated that the new messages were delivered, and showed up in the archive. I shut down Mailman again. I had the database team restore the Mailman database from the backup. I started Mailman I logged into Mailman I was expecting to see the messages in the archive prior to the backup and all the messages after the backup would be gone. When went into the list archive, I get a spinning icon and no messages showed up. I quit, restarted my browser, and logged in again, and the same thing. I then sent a test message to the list and the message was delivered to the list. Then when I went to check the archive, the spinning icon was gone and that test message was there along with all the messages before the backup was taken.
Is this normal? After a restore did the new message cause a refresh of archived messages etc? Is there a way to do this without sending a message to a list after a restoration of the database?
Are there better steps I should take when restoring the database?
Thanks
On 8/5/22 09:38, bob B via Mailman-users wrote:
I have a test Mailman 3 environment. We are using docker, and we are running the database outside of docker in mysql. I need to perform a DR recovery test.
I shut down Mailman in docker (mailman-core & mailman-web, the only docker images I have running for Mailman).
Does shutting down mailman-web also shut down Django qcluster? If not, this could be an issue.
While Mailman was stopped, I had our database team take a backup dump of the database. I then restarted Mailman in docker. I threw some more messages at a test list. I validated that the new messages were delivered, and showed up in the archive. I shut down Mailman again. I had the database team restore the Mailman database from the backup. I started Mailman I logged into Mailman I was expecting to see the messages in the archive prior to the backup and all the messages after the backup would be gone.
I would expect that too.
When went into the list archive, I get a spinning icon and no messages showed up.
I don't offhand know what the issue would be unless Django qcluster was running the whole time.
I quit, restarted my browser, and logged in again, and the same thing. I then sent a test message to the list and the message was delivered to the list. Then when I went to check the archive, the spinning icon was gone and that test message was there along with all the messages before the backup was taken.
Possibly one of the Django periodic jobs, maybe cache_cleanup or daily_cleanup would fix it without a message.
Is this normal? After a restore did the new message cause a refresh of archived messages etc? Is there a way to do this without sending a message to a list after a restoration of the database?
Are there better steps I should take when restoring the database?
I don't know what better you could do as far as the save and restore are concerned as long as shutting down mailman-web stops qcluster.
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark,
Thanks, I also would assume the Django cluster runs in the mailman-web, and during my test mailman-core & mailman-web, docker images were down, so the whole environment was shut down.
Is there a way to rebuild the archives for lists? What about rebuilding any static web pages outside the database, is that even necessary? I know in a real disaster the environment will not be down, so I see three scenarios:
Rebuild my docker environment and point it to the database if the docker environment had a disaster. Point the docker environment to a restored database, if the disaster just happened to the database. Rebuild my docker environment and restore the database from a backup, if they both had disasters.
bob B via Mailman-users writes:
Is there a way to rebuild the archives for lists?
If you keep a backup as an mbox file (this is what the "sample archiver" does), you can always reinitialize HyperKitty and reimport as if you were migrating a Pipermail archive from Mailman 2. HyperKitty (and Postorius) have data stores that are independent of those used by Mailman core.
What about rebuilding any static web pages outside the database, is that even necessary?
If you have changed any templates, you would need to reproduce those changes. Otherwise it's presumably just a matter of reinstalling the Django apps and running the "collectstatic" command in the Django manager.
I know in a real disaster the environment will not be down, so I see three scenarios:
They seem plausible (ie, covering all possibilities and the solutions) to me.
Stephen,
Thanks, but how would make backup mbox files in mailman3 with Hyperkitty? I thought all the messages were stored in the database and there were no mbox files anymore. I assume I would have to set up an automated mbox export? But this seems like an extra step in preparing for disasters. I would assume I would just be able to restore the database and reinstall/rebuild mailman and then run maybe a script to make sure the archives are rebuilt? and then I would be good to go.
bob B via Mailman-users writes:
Thanks, but how would make backup mbox files in mailman3 with Hyperkitty?
You don't do it with HyperKitty (I'm pretty sure you can export mboxes from HyperKitty, but I don't think it would be timely, you have to do it by hand). The sample archiver (which you can enable in mailman.cfg IIRC, haven't done it in a while) would make the mbox files directly from the core at the same time that they are sent to HyperKitty. It's possible to have both HyperKitty and the sample archiver enabled at the same time.
I thought all the messages were stored in the database and there were no mbox files anymore.
That's true in HyperKitty. The sample archiver is a separate facility that's part of core. All it does is append to an mbox on the core's host. It has no facility for display to users. It's basically a developer demo for how you attach an archiver to Mailman core, but it's also a possible backup approach for a very careful admin like you.
I would assume I would just be able to restore the database
True, but how frequently do you back up the database? The "also store to mbox" approach I described loses no messages even if your database gets trashed (unless the mbox also gets trashed in the the disaster, but if I understand your setup correctly, the mbox would be on the core's host, while the database server and storage is on a different host, so trashing both would be a really bad situation, hopefully very unlikely).
and reinstall/rebuild mailman and then run maybe a script to make sure the archives are rebuilt?
If the database restore is successful, there's no rebuilding to do. The database is the archive. You just need to reattach it to HyperKitty.
and then I would be good to go.
Yes, modulo the period between most recent database backup and disaster.
Regards, Steve
On 8/9/22 07:19, Stephen J. Turnbull wrote:
You don't do it with HyperKitty (I'm pretty sure you can export mboxes from HyperKitty, but I don't think it would be timely, you have to do it by hand). The sample archiver (which you can enable in mailman.cfg IIRC, haven't done it in a while) would make the mbox files directly from the core at the same time that they are sent to HyperKitty. It's possible to have both HyperKitty and the sample archiver enabled at the same time.
The sample archiver is actually called the prototype archiver and it
is typically enabled by default.
The messages are stored in maildir format in Mailman's var/archives/prototype/ directory in sub-directories with names of the list posting addresses. You would need to convert maildir to mbox format to import to hyperkitty. A simple script to do this is
#!/usr/bin/python3
"""Run this with the list posting address as it's argument."""
import os
import sys
from mailbox import mbox, Maildir
MDIR = '/path/to/mailman/var/archives/prototype/'
MBOX = '/path/to/output_directory'
if len(sys.argv) != 2:
print(f'Usage: {sys.argv[0]} list_posting_address', file=sys.stderr)
sys.exit(1)
mdir = Maildir(os.path.join(MDIR, sys.argv[1]), create=False)
mlbox = mbox(os.path.join(MBOX, sys.argv[1]), create=True)
for msg in mdir:
mlbox.add(msg)
mdir.close()
mlbox.close()
-- Mark Sapiro mark@msapiro.net The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (3)
-
bob B -
Mark Sapiro -
Stephen J. Turnbull