Hyperkitty import from mailman2
Hi
I have installed mailman3 - it looks very great!
Now I wanted to try import from mailman2 mbox, so i did:
python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox
It shows that it have sucessfully imported all emails and I can see them from database (sqlite3), but I cannot see them from GUI (hyperkitty). Only thing I can see is this inactive "test" list.
I have tried that I create list before importing, still same result.
All new emails are archiving to another lists (which are not imported, but new installations).
Any help is appreciated!
All the best Sten
Now I wanted to try import from mailman2 mbox, so i did:
python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox
It shows that it have sucessfully imported all emails and I can see them from database (sqlite3), but I cannot see them from GUI (hyperkitty). Only thing I can see is this inactive "test" list.
I have tried that I create list before importing, still same result.
I have done this several times for several different lists and it has always worked with no problems. I have always created the list first.
I do not use any of the options other than -l. My settings location is built in to the django admin script I run. So what I run is equivalent to
python2 manage.py hyperkitty_import --settings .. -l .. mbox
Do you have cron set up to run the periodic django tasks as in
@hourly python2 manage.py runjobs --settings settings hourly @daily python2 manage.py runjobs --settings settings daily @weekly python2 manage.py runjobs --settings settings weekly @monthly python2 manage.py runjobs --settings settings monthly @yearly python2 manage.py runjobs --settings settings yearly 0,15,30,45 * * * * python2 manage.py runjobs --settings settings quarter_hourly
- python2 manage.py runjobs --settings settings minutely
If not, did you run
python2 manage.py update_index --settings settings
after the hyperkitty_import?
I'm actually not sure any of this will help since it should only affect the search index, not the message display.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On 12/22/2017 03:39 PM, Sten Aus wrote:
Hi
I have installed mailman3 - it looks very great!
Now I wanted to try import from mailman2 mbox, so i did:
python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox
It shows that it have sucessfully imported all emails and I can see them from database (sqlite3), but I cannot see them from GUI (hyperkitty). Only thing I can see is this inactive "test" list.
How did you try to access them using Hyperkitty? Can you try doing that while logged in as a django superuser? Could you try using a permalink for a message? If you can access the files from the database you should be able to construct the urls using the list name and the message_id_hash of a message. You should try this using DEBUG=True in the settings.py. The URL would be similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/M...
Another option would be to check if you can access the data using a python shell.
python manage.py shell to open a python shell preconfigured for
Hyperkitty and then input the following two lines:
from hyperkitty.models.email import Email Email.objects.all()
This should print out all the messages that are stored in the database. (You can limit them by appending "[10:]", or any other python slice)
What do you see in the UI? Is it somewhat similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org ?
Hi
Thank you Mark and Simon for your input!
I can see imported list from Hyperkitty GUI, but it's inactive. I can confirm that I can see those emails from command line (python shell shows Email Objects).
Import shows everything is OK:
# python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox Only emails after 2010-01-01 00:00:00+02:00 will be imported Importing from mbox file /tmp/test.mbox to test@list.domain.com /--some 22 emails being imported here --/ 22 emails read 22 email added to the database 22 emails are stored into the database Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index will be updated every minute. Run the 'manage.py runjob update_index' command to update it now.
# python2 manage.py update_index --settings settings Indexing 50 emails NB! Only 6 of them are archived from new list (which has been created from Apache). I have imported 2 times 22 emails from /tmp/test.mbox file (to different lists).
I am using Apache2.4 with mod wsgi and Postorius and Hyperkitty are in the same django application. Also tested with running server directly and then connecting to a port 8000, but I guess it does not matter, because both of them are using Python script.
I have not yet managed to set up cron tasks, as this is relatively new installation, but I have ran them manually. Also, I have done update_index (runjob update_index and manage.py update_index, both ways).
SQLite database shows emails, for example there's a thread FMXQVFDRQVRMINVXI4DJB4SHYTJ3EPWP, but If I insert it to web URL I get Django error: No Thread matches the given query.
As Postorius and Hyperkitty are in the same Django, they share sqlite database (mailman/web/database.db). Mailman has it's own database in mailman/var/data/mailman.db.
Happy holidays Sten
On 22/12/2017 21:09, Simon Hanna wrote:
On 12/22/2017 03:39 PM, Sten Aus wrote:
Hi
I have installed mailman3 - it looks very great!
Now I wanted to try import from mailman2 mbox, so i did:
python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox
It shows that it have sucessfully imported all emails and I can see them from database (sqlite3), but I cannot see them from GUI (hyperkitty). Only thing I can see is this inactive "test" list.
How did you try to access them using Hyperkitty? Can you try doing that while logged in as a django superuser? Could you try using a permalink for a message? If you can access the files from the database you should be able to construct the urls using the list name and the message_id_hash of a message. You should try this using DEBUG=True in the settings.py. The URL would be similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/M...
Another option would be to check if you can access the data using a python shell.
python manage.py shellto open a python shell preconfigured for Hyperkitty and then input the following two lines:from hyperkitty.models.email import Email Email.objects.all()
This should print out all the messages that are stored in the database. (You can limit them by appending "[10:]", or any other python slice)
What do you see in the UI? Is it somewhat similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org ?
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Just a suggestion -- if your list didn't have any e-mails in the last month, the front page will show it as inactive. Dig into the archive links on the left side to actually see your e-mails. I've been bitten by that behavior as well. The front page simply doesn't show e-mails older than a month.
Look here: https://lists.secretchronicles.org/hyperkitty/list/tsc-devel@lists.secretchr...
No emails on the front page. Now go here: https://lists.secretchronicles.org/hyperkitty/list/tsc-devel@lists.secretchr...
Well there.
Greetings Marvin
Am 23. December 2017 um 01:05 Uhr +0200 schrieb Sten Aus <sten.aus@eenet.ee>:
Hi
Thank you Mark and Simon for your input!
I can see imported list from Hyperkitty GUI, but it's inactive. I can confirm that I can see those emails from command line (python shell shows Email Objects).
Import shows everything is OK:
# python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox Only emails after 2010-01-01 00:00:00+02:00 will be imported Importing from mbox file /tmp/test.mbox to test@list.domain.com /--some 22 emails being imported here --/ 22 emails read 22 email added to the database 22 emails are stored into the database Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index will be updated every minute. Run the 'manage.py runjob update_index' command to update it now.
# python2 manage.py update_index --settings settings Indexing 50 emails NB! Only 6 of them are archived from new list (which has been created from Apache). I have imported 2 times 22 emails from /tmp/test.mbox file (to different lists).
I am using Apache2.4 with mod wsgi and Postorius and Hyperkitty are in the same django application. Also tested with running server directly and then connecting to a port 8000, but I guess it does not matter, because both of them are using Python script.
I have not yet managed to set up cron tasks, as this is relatively new installation, but I have ran them manually. Also, I have done update_index (runjob update_index and manage.py update_index, both ways).
SQLite database shows emails, for example there's a thread FMXQVFDRQVRMINVXI4DJB4SHYTJ3EPWP, but If I insert it to web URL I get Django error: No Thread matches the given query.
As Postorius and Hyperkitty are in the same Django, they share sqlite database (mailman/web/database.db). Mailman has it's own database in mailman/var/data/mailman.db.
Happy holidays Sten
On 22/12/2017 21:09, Simon Hanna wrote:
On 12/22/2017 03:39 PM, Sten Aus wrote:
Hi
I have installed mailman3 - it looks very great!
Now I wanted to try import from mailman2 mbox, so i did:
python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox
It shows that it have sucessfully imported all emails and I can see them from database (sqlite3), but I cannot see them from GUI (hyperkitty). Only thing I can see is this inactive "test" list.
How did you try to access them using Hyperkitty? Can you try doing that while logged in as a django superuser? Could you try using a permalink for a message? If you can access the files from the database you should be able to construct the urls using the list name and the message_id_hash of a message. You should try this using DEBUG=True in the settings.py. The URL would be similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/M...
Another option would be to check if you can access the data using a python shell.
python manage.py shellto open a python shell preconfigured for Hyperkitty and then input the following two lines:from hyperkitty.models.email import Email Email.objects.all()
This should print out all the messages that are stored in the database. (You can limit them by appending "[10:]", or any other python slice)
What do you see in the UI? Is it somewhat similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org ?
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
-- Blog: https://www.guelkerdev.de PGP/GPG ID: F1D8799FBCC8BC4F
Oh my god. :) That's it! There are no latest messages and front page shows only last month "activity". Thank you! I did not thought that I should check some older posts. :D
So, import is working just fine, I guess! ;)
But how did you manage to import all the lists and members from Mailman2 to Mailman3? As I understand Mailman3 does not "support" (by manual) import right now?
On 24/12/2017 00:11, Marvin Gülker wrote:
Just a suggestion -- if your list didn't have any e-mails in the last month, the front page will show it as inactive. Dig into the archive links on the left side to actually see your e-mails. I've been bitten by that behavior as well. The front page simply doesn't show e-mails older than a month.
Look here: https://lists.secretchronicles.org/hyperkitty/list/tsc-devel@lists.secretchr...
No emails on the front page. Now go here: https://lists.secretchronicles.org/hyperkitty/list/tsc-devel@lists.secretchr...
Well there.
Greetings Marvin
Am 23. December 2017 um 01:05 Uhr +0200 schrieb Sten Aus <sten.aus@eenet.ee>:
Hi
Thank you Mark and Simon for your input!
I can see imported list from Hyperkitty GUI, but it's inactive. I can confirm that I can see those emails from command line (python shell shows Email Objects).
Import shows everything is OK:
# python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox Only emails after 2010-01-01 00:00:00+02:00 will be imported Importing from mbox file /tmp/test.mbox to test@list.domain.com /--some 22 emails being imported here --/ 22 emails read 22 email added to the database 22 emails are stored into the database Computing thread structure Synchronizing properties with Mailman Warming up cache The full-text search index will be updated every minute. Run the 'manage.py runjob update_index' command to update it now.
# python2 manage.py update_index --settings settings Indexing 50 emails NB! Only 6 of them are archived from new list (which has been created from Apache). I have imported 2 times 22 emails from /tmp/test.mbox file (to different lists).
I am using Apache2.4 with mod wsgi and Postorius and Hyperkitty are in the same django application. Also tested with running server directly and then connecting to a port 8000, but I guess it does not matter, because both of them are using Python script.
I have not yet managed to set up cron tasks, as this is relatively new installation, but I have ran them manually. Also, I have done update_index (runjob update_index and manage.py update_index, both ways).
SQLite database shows emails, for example there's a thread FMXQVFDRQVRMINVXI4DJB4SHYTJ3EPWP, but If I insert it to web URL I get Django error: No Thread matches the given query.
As Postorius and Hyperkitty are in the same Django, they share sqlite database (mailman/web/database.db). Mailman has it's own database in mailman/var/data/mailman.db.
Happy holidays Sten
On 22/12/2017 21:09, Simon Hanna wrote:
On 12/22/2017 03:39 PM, Sten Aus wrote:
Hi
I have installed mailman3 - it looks very great!
Now I wanted to try import from mailman2 mbox, so i did:
python2 manage.py hyperkitty_import --settings settings --ignore-mtime -v 3 --since 2010-01-01 -l test@list.domain.com /tmp/test.mbox
It shows that it have sucessfully imported all emails and I can see them from database (sqlite3), but I cannot see them from GUI (hyperkitty). Only thing I can see is this inactive "test" list.
How did you try to access them using Hyperkitty? Can you try doing that while logged in as a django superuser? Could you try using a permalink for a message? If you can access the files from the database you should be able to construct the urls using the list name and the message_id_hash of a message. You should try this using DEBUG=True in the settings.py. The URL would be similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org/message/M...
Another option would be to check if you can access the data using a python shell.
python manage.py shellto open a python shell preconfigured for Hyperkitty and then input the following two lines:from hyperkitty.models.email import Email Email.objects.all()
This should print out all the messages that are stored in the database. (You can limit them by appending "[10:]", or any other python slice)
What do you see in the UI? Is it somewhat similar to http://lists.mailman3.org/archives/list/mailman-users@mailman3.org ?
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
On 12/24/2017 12:30 PM, Sten Aus wrote:
But how did you manage to import all the lists and members from Mailman2 to Mailman3? As I understand Mailman3 does not "support" (by manual) import right now?
Mailman's 'bin/mailman import21' is the tool for this. It has a few issues. It doesn't import DMARC mitigation settings and I've seen one case out of several where it failed to import the list membership.
I also once saw and reported <https://gitlab.com/mailman/mailman/issues/270>, but I've imported several lists subsequently without seeing that issue.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Am 24. December 2017 um 22:30 Uhr +0200 schrieb Sten Aus <sten.aus@eenet.ee>:
Oh my god. :) That's it! There are no latest messages and front page shows only last month "activity". Thank you! I did not thought that I should check some older posts. :D
I know that feeling. It took me a while to figure this out as well. Error on OSI Layer 8.
But how did you manage to import all the lists and members from Mailman2 to Mailman3? As I understand Mailman3 does not "support" (by manual) import right now?
You'll be surprised, but those mailinglists were not managed by Mailman2 before. They were managed by mlmmj[1]. I wrote a script that converted mlmmj's archive into mbox format, and then used the "hyperkitty_import" command as explained by the Hyperkitty manual[2]. This worked surprisingly well. As for the subscribers, Mailman 3's Postorius has a mass-subscription facility in the admin interface. I extracted the list of all subscriber emails from the mlmmj mailinglist and used that mass-subscription facility to subscribe people again. There's an option to disable delivery of the welcome email that should be ticked before this I guess, though. It can be reset afterwards again.
Greetings Marvin
[1] http://mlmmj.org/ [2] http://hyperkitty.readthedocs.io/en/latest/install.html#importing-the-curren...
-- Blog: https://www.guelkerdev.de PGP/GPG ID: F1D8799FBCC8BC4F
Hi
Happy new year and thank you for your support!
I have managed to get it to work. I have imported config.pck from Mailman 2.1 and list archive from mobx format and it shows just great! Now just little more testing and I can finally import all old data to new Mailman 3.
I imported list called "test" and the list was already created inside Mailman3 (before import). And import was successful.
I will let you know the results!
All the best Sten
On 25/12/2017 10:25, Marvin Gülker wrote:
Am 24. December 2017 um 22:30 Uhr +0200 schrieb Sten Aus <sten.aus@eenet.ee>:
Oh my god. :) That's it! There are no latest messages and front page shows only last month "activity". Thank you! I did not thought that I should check some older posts. :D I know that feeling. It took me a while to figure this out as well. Error on OSI Layer 8.
But how did you manage to import all the lists and members from Mailman2 to Mailman3? As I understand Mailman3 does not "support" (by manual) import right now? You'll be surprised, but those mailinglists were not managed by Mailman2 before. They were managed by mlmmj[1]. I wrote a script that converted mlmmj's archive into mbox format, and then used the "hyperkitty_import" command as explained by the Hyperkitty manual[2]. This worked surprisingly well. As for the subscribers, Mailman 3's Postorius has a mass-subscription facility in the admin interface. I extracted the list of all subscriber emails from the mlmmj mailinglist and used that mass-subscription facility to subscribe people again. There's an option to disable delivery of the welcome email that should be ticked before this I guess, though. It can be reset afterwards again.
Greetings Marvin
[1] http://mlmmj.org/ [2] http://hyperkitty.readthedocs.io/en/latest/install.html#importing-the-curren...
participants (4)
-
Mark Sapiro -
Marvin Gülker -
Simon Hanna -
Sten Aus