On Thu, Apr 17, 2025 at 3:14 PM Christian Schneider < christian.schneider@tu-dortmund.de> wrote:

Thank you for your instant reply!

Mihai Moldovan wrote:

...

...

But I could find no information about the location where hyperkitty stores the archives on an Ubuntu-/Linux-system. I assume the archives are stored underneath "/var/..." this is only a guess. But is this guess right? Not quite. To the best of my knowledge, by default, a venv installation is fully contained under /opt/mailman, with archives and stuff stored at /opt/mailman/var/lib/...

This information is very helpful! Actually I have about 45GB of archives and these archives only hold messages of the last year... I run a clean-up script every first day of a month which deletes posts older than 365 days... So there is lot of disk space needed... ;-) (and even more if I plan to keep posts around for a longer time)

...

...

Another question is: How much space do I have to plan (roughly) for the "/opt"-partition for the mailman install if the archives are stored under "/var/..."? Depends on your archives, the way things are archived (you can choose different database backends, and these might make a difference in how efficiently or inefficiently data is stored) and whether you want to use the full-text search index, which adds another big chunk of data. I can give you a few pointers regarding my setup: my mailman 2.1 data directory takes up about 8.2 GB of disk space, most of which are archives, naturally (8.1 GB). I've imported most of the big archives to mailman 3 already, with a full-text search index, and that uses 5.2 GB disk space currently, but I'm still missing a few archives, so that'll still grow a bit. It will probably end up at about the same size, but I can come back to that once my import is fully done (which sadly will take a few days). Now, the 8.1 GB of mailman 2.1 archives are stored pretty inefficiently, because they are essentially duplicated (as a private mbox archive and public text archives). Mailman 3 doesn't do that - at least not directly - but instead stores archives once into the database, and queries the database whenever the archives are accessed via hyperkitty, so I would *assume* the size you need for the mailman 3 data after a successful migration is just a bit shy of what you needed for the mailman 2.1 data, maybe even a bit less.

Perfect! :-) If I mount an extra partition of 250Gigs at /opt this should carry me for a longer time... And - if I setup a logical volume with LVM - there is an option to stack up the space with ease... ;-)

Does your DB backend also store the data files in /opt? I doubt it. The archives will be stored in the DB backend. So unless you use SQLite backend, the data will go where you configure your DB to store data- whether it's MariaDB or PostgreSQL.

-- Best regards, Odhiambo WASHINGTON, Nairobi,KE +254 7 3200 0004/+254 7 2274 3223 In an Internet failure case, the #1 suspect is a constant: DNS. "Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-) [How to ask smart questions: http://www.catb.org/~esr/faqs/smart-questions.html]

Sorry. Didn't leave proper spacing after the original quote.

My complete reply

The Virtualenv Installation guide has you install PostgreSQL as the database backend and HyperKitty stores attachments in the database as blobs. The default data directory for PostgreSQL in Ubuntu 24.04 LTS is located at: /var/lib/postgresql/16/main

However, if you wanted you could always change the location before initializing the database by updating the setting in the postgresql.conf file located at: /etc/postgresql/16/main/postgresql.conf

Another option could be to use symlinks. Move the data directory to a location of choice and then create a symlink in the original location that points to the new/actual location.

• On 4/17/25 18:47, German Rodriguez wrote:

I wondered about the difference of the two archivers (prototype/hyperkitty) and whether both should be enabled.

Mark explains here that the prototype archiver is for storing the raw email message, whereas HyperKitty only stores select information. https://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/...

Interesting. I probably want the prototype archiver enabled, even if that means duplicating most of the archives and doubling the amount of data stored.

I was not aware that Hyperkitty doesn't store all information. I'd like to have all information archived (in order to be able to restore it again, if need be), even if they aren't published.

Now... is there a way to import mbox files into the prototype archiver? Looking around, I haven't found anything mentioning this. Since it's just using the Maildir format, I guess I could find a tool that unpacks messages from an mbox to a Maildir destination and get that done, but if mailman has some command to do that, it would be even better.

Mihai

• On 4/17/25 19:38, Mark Sapiro wrote:

There is no Mailman command to do this, but a Python script is very simple. Something like

[script]

Great, that will definitely create it in the format expected by Python and hence mailman! Thank you very much.

Mihai

On 4/17/25 11:35 AM, Mihai Moldovan wrote:

New incoming messages will have the correct data, of course, but imported ones wouldn't, so I'll have to use a more sophisticated approach to handle them, probably by going through mailman's email wrapper, figuring out how to generate a msgdata object for a message, using RFC2369.process and maybe more.

Here's an example script. You need to run this with /opt/mailman/mm/venv/bin/python to get access to the mailman imports.

from mailbox import Maildir, mbox
from mailman.email.message import Message
from mailman.handlers.rfc2369 import process
from mailman.interfaces.listmanager import IListManager
from mailman.utilities.email import add_message_hash
from zope.component import getUtility
mb = mbox('path/to/mbox', factory=Message, create=False)
md = Maildir('path/to/maildir', create=False)
mlist = getUtility(IListManager).get_by_list_id('your.list.id')
for msg in mb:
     add_message_hash(msg)
     process(mlist, msg, {})
     md.add(msg)
mb.close()
md.close()

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan