Recommended way to backup a mailman3 instance
Hi,
I'd like to schedule periodic backup for our mailman3 instance. Are there recommended ways for doing this without down times?
Thanks in advance,
_g.
Gilles Filippini a écrit le 10/04/2020 à 10:39 :
Hi,
I'd like to schedule periodic backup for our mailman3 instance. Are there recommended ways for doing this without down times?
I should have precised that our mailman3 instance is dockerized. Then, as I understand it, these backups should be sufficient:
- MailmanCore's volume (/opt/mailman/var folder)
- Database dump
Anything else?
Thanks,
_g.
On Sun, Apr 12, 2020, at 7:23 AM, Gilles Filippini wrote:
Gilles Filippini a écrit le 10/04/2020 à 10:39 :
Hi,
I'd like to schedule periodic backup for our mailman3 instance. Are there recommended ways for doing this without down times?
I should have precised that our mailman3 instance is dockerized. Then, as I understand it, these backups should be sufficient:
- MailmanCore's volume (/opt/mailman/var folder)
/opt/mailman/core/var
- Database dump
Anything else?
No, that should be enough. Mailman-web is fairly stateless and stores everything except the settings and logs in the database. If you have settings version controlled then you don't need anything else. There is also fulltext_index, but that doesn't need to be backed up because it can be easy re-generated with a single django-admin command.
Mailman core's entire state is in the var/ directory above.
Thanks,
_g.
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
Abhilash Raj a écrit le 12/04/2020 à 19:34 :
On Sun, Apr 12, 2020, at 7:23 AM, Gilles Filippini wrote:
Gilles Filippini a écrit le 10/04/2020 à 10:39 :
Hi,
I'd like to schedule periodic backup for our mailman3 instance. Are there recommended ways for doing this without down times?
I should have precised that our mailman3 instance is dockerized. Then, as I understand it, these backups should be sufficient:
- MailmanCore's volume (/opt/mailman/var folder)
/opt/mailman/core/var
- Database dump
Anything else?
No, that should be enough. Mailman-web is fairly stateless and stores everything except the settings and logs in the database. If you have settings version controlled then you don't need anything else. There is also fulltext_index, but that doesn't need to be backed up because it can be easy re-generated with a single django-admin command.
I don't know django. Would you mind giving some pointers about this django-admin command to re-regenerate fulltext_index?
Mailman core's entire state is in the var/ directory above.
Ack. Thanks,
_g,
Abhilash Raj wrote:
Mailman core's entire state is in the var/ directory above.
I was working round my own backup strategy a few weeks ago as I discussed earlier on this list. One thing I found when the directory structure that Mailman Core uses is that the lists directories do not get re-created when Mailman Core starts if they don't already exist. Is this something that should happen, and when would those directories get created? Every other directory did get re-created.
In my case I was moving from a Dockerised installation to one using venvs, and as the directories from the Docker installation were empty as there hadn't been any postings for a few days, I didn't transfer those over. I created the list directories myself and everything carried on.
Thanks. Andrew.
On 4/13/20 3:22 AM, Andrew Hodgson wrote:
I was working round my own backup strategy a few weeks ago as I discussed earlier on this list. One thing I found when the directory structure that Mailman Core uses is that the lists directories do not get re-created when Mailman Core starts if they don't already exist. Is this something that should happen, and when would those directories get created? Every other directory did get re-created.
In my case I was moving from a Dockerised installation to one using venvs, and as the directories from the Docker installation were empty as there hadn't been any postings for a few days, I didn't transfer those over. I created the list directories myself and everything carried on.
These var/lists/<list-id>/ directories hold the digest.mmdf files which are mailboxes containing the messages for the next digest. A list post will be added to the digest.mmdf file which will be created if necessary, but if the var/lists/<list-id>/ directory doesn't exist, the create will fail.
Thus, the var/lists/<list-id>/ directories need to exist. Normally, they are created when the list is created so they need to be part of a backup.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mon, Apr 13, 2020, at 9:40 AM, Mark Sapiro wrote:
On 4/13/20 3:22 AM, Andrew Hodgson wrote:
I was working round my own backup strategy a few weeks ago as I discussed earlier on this list. One thing I found when the directory structure that Mailman Core uses is that the lists directories do not get re-created when Mailman Core starts if they don't already exist. Is this something that should happen, and when would those directories get created? Every other directory did get re-created.
In my case I was moving from a Dockerised installation to one using venvs, and as the directories from the Docker installation were empty as there hadn't been any postings for a few days, I didn't transfer those over. I created the list directories myself and everything carried on.
These var/lists/<list-id>/ directories hold the digest.mmdf files which are mailboxes containing the messages for the next digest. A list post will be added to the digest.mmdf file which will be created if necessary, but if the var/lists/<list-id>/ directory doesn't exist, the create will fail.
Thus, the var/lists/<list-id>/ directories need to exist. Normally, they are created when the list is created so they need to be part of a backup.
If they are empty, perhaps the digest creation process should just re-create them instead of failing to create/send digests? We don't really need the directory structure to exist.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mailman-users mailing list -- mailman-users@mailman3.org To unsubscribe send an email to mailman-users-leave@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- thanks, Abhilash Raj (maxking)
On 4/17/20 4:51 PM, Abhilash Raj wrote:
If they are empty, perhaps the digest creation process should just re-create them instead of failing to create/send digests? We don't really need the directory structure to exist.
I created <https://gitlab.com/mailman/mailman/-/issues/699> for this.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Mark Sapiro wrote:
On 4/17/20 4:51 PM, Abhilash Raj wrote:
If they are empty, perhaps the digest creation process should just re-create them instead of failing to create/send digests? We don't really need the directory structure to exist.
I created <https://gitlab.com/mailman/mailman/-/issues/699> for this.
In my situation I noticed the directories weren't created very quickly as my Exim config looks for the list directories to identify whether that address exists. I don't know how Postfix does the aliasing but for my case it would make sense to get the directories created as soon as possible, not sure if the digest handler would be soon enough or not.
Thanks. Andrew.
Andrew Hodgson a écrit le 18/04/2020 à 12:26 :
Mark Sapiro wrote:
On 4/17/20 4:51 PM, Abhilash Raj wrote:
If they are empty, perhaps the digest creation process should just re-create them instead of failing to create/send digests? We don't really need the directory structure to exist.
I created <https://gitlab.com/mailman/mailman/-/issues/699> for this.
In my situation I noticed the directories weren't created very quickly as my Exim config looks for the list directories to identify whether that address exists. I don't know how Postfix does the aliasing but for my case it would make sense to get the directories created as soon as possible, not sure if the digest handler would be soon enough or not.
There is no need for these directories on the Exim side actually. You can safely drop the related checks from the Exim configuration [1].
[1] https://github.com/maxking/docker-mailman/pull/371
_g.
Gilles Filippini wrote:
Andrew Hodgson a écrit le 18/04/2020 à 12:26 :
[...]
In my situation I noticed the directories weren't created very quickly as my Exim config looks for the list directories to identify whether that address exists. I don't know how Postfix does the aliasing but for my case it would make sense to get the directories created as soon as possible, not sure if the digest handler would be soon enough or not.
There is no need for these directories on the Exim side actually. You can safely drop the related checks from the Exim configuration [1].
Interesting, if you drop these checks does the mail get rejected at SMTP time or does Exim need to generate a bounce after it tries to deliver the message via LMTP?
What happens if the Mailman LMTP is unavailable at the point when the message comes in? In the current environment it gets queued, but that happens because Exim knows the list exists.
Reducing backscatter is quite a high priority for me as I get a lot of email to junk aliases at the lists domain which currently get rejected at the envelope stage.
Thanks. Andrew.
Andrew Hodgson a écrit le 18/04/2020 à 12:51 :
Gilles Filippini wrote:
Andrew Hodgson a écrit le 18/04/2020 à 12:26 :
[...]
In my situation I noticed the directories weren't created very quickly as my Exim config looks for the list directories to identify whether that address exists. I don't know how Postfix does the aliasing but for my case it would make sense to get the directories created as soon as possible, not sure if the digest handler would be soon enough or not.
There is no need for these directories on the Exim side actually. You can safely drop the related checks from the Exim configuration [1].
Interesting, if you drop these checks does the mail get rejected at SMTP time or does Exim need to generate a bounce after it tries to deliver the message via LMTP?
As I understand it from the logs, Exim forwards the LMTP bounce to the sender.
What happens if the Mailman LMTP is unavailable at the point when the message comes in? In the current environment it gets queued, but that happens because Exim knows the list exists.
No, Exim doesn't know the list exists. It just knows that it is allowed to go through the mailman transport. Then I guess this is the same behavior when the checks are disabled, except that in this case junk email is queued as well, provided it targets one of your lists domain.
Reducing backscatter is quite a high priority for me as I get a lot of email to junk aliases at the lists domain which currently get rejected at the envelope stage.
Then this setting might no be what you want. I set it up because my services are dockerized and I'd prefer not to mount the mailman-core volume into the Exim container.
_g.
On 4/18/20 3:26 AM, Andrew Hodgson wrote:
In my situation I noticed the directories weren't created very quickly as my Exim config looks for the list directories to identify whether that address exists. I don't know how Postfix does the aliasing but for my case it would make sense to get the directories created as soon as possible, not sure if the digest handler would be soon enough or not.
The directory is created when the list is created and should always be there if the list is there. The idea behind <https://gitlab.com/mailman/mailman/-/issues/699> is only to correct a situation where the directory for a list is absent because it was deleted or the server was restored from a backup that didn't contain it.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
participants (4)
-
Abhilash Raj -
Andrew Hodgson -
Gilles Filippini -
Mark Sapiro