Recently active discussions - Hyperkitty
Hi,
Sometimes not sure if it's preferable to open a gitlab issue, or chat on the mailing list.
I have run a large archive import into mailman3/hyperkitty. After that's done, waited a day, for scheduled cron jobs to complete. In "All Threads", all the email archives are present. The import worked. In "RECENTLY ACTIVE DISCUSSIONS" on the main page, it says:
RECENTLY ACTIVE DISCUSSIONS No discussions this month (yet).
Nothing there. That means if I direct users to visit the site, the page, it appears empty. That's too bad. :-) It should display recent emails.
What would affect this feature during an archive import? Is it a problem with the import script itself, or with a scheduled cron job, or something else? Would you expect "Recently active discussions" be empty when very recent archives do exist?
Thanks, Sam
On 5/31/24 12:22, Sam Darwin wrote:
Nothing there. That means if I direct users to visit the site, the page, it appears empty. That's too bad. :-) It should display recent emails.
What would affect this feature during an archive import? Is it a problem with the import script itself, or with a scheduled cron job, or something else? Would you expect "Recently active discussions" be empty when very recent archives do exist?
It's based on the Date: headers of the messages. Are there messages with Date: headers in the last month?
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Are there messages with Date: headers in the last month?
Well, there are definitely many messages from the last month.
Hyperkitty, for each message, shows in the upper-right hand corner of the message "30 May 2024" "29 May 2024" "20 May 2024 3:00pm" and so on. Each message has a date shown.
"Threads by Month" is populated with messages from the last month.
"All Threads" is populated with messages from the last month.
However, "Recently Active Discussions" is empty.
That is not conclusive regarding "Date: headers". But wouldn't "Threads by Month" use the same date information?
By the way, during the import "Message 04b901d993ca$d6cf3180$846d9480$@gmail.com failed to import, skipping" and "psycopg2.OperationalError: SSL connection has been closed unexpectedly".
However, the import continues past those errors. For example:
Computing thread structure Synchronizing properties with Mailman INFO hyperkitty.lib.mailman 10820 emails left to refresh, checked 1000 10820 emails left to refresh, checked 1000 INFO hyperkitty.lib.mailman 10820 emails left to refresh, checked 2000 10820 emails left to refresh, checked 2000 INFO hyperkitty.lib.mailman 10820 emails left to refresh, checked 3000 10820 emails left to refresh, checked 3000
It all eventually completes with exit code 0. Generally, successful. So it's unclear if a skipped message is the cause.
On 5/31/24 13:06, Sam Darwin wrote:
Are there messages with Date: headers in the last month?
Well, there are definitely many messages from the last month.
Hyperkitty, for each message, shows in the upper-right hand corner of the message "30 May 2024" "29 May 2024" "20 May 2024 3:00pm" and so on. Each message has a date shown.
Those dates should be the ones used for recent threads.
"Threads by Month" is populated with messages from the last month. "All Threads" is populated with messages from the last month. However, "Recently Active Discussions" is empty.
That is not conclusive regarding "Date: headers". But wouldn't "Threads by Month" use the same date information?
Yes.
Perhaps the hyperkitty recent_threads_cache job can fix this, but it is one of the jobs run daily by cron (assuming cron is running them) so after 24 hours, it should have run.
By the way, during the import "Message 04b901d993ca$d6cf3180$846d9480$@gmail.com failed to import, skipping" and "psycopg2.OperationalError: SSL connection has been closed unexpectedly".
I'm guessing that was one message reported on two lines with the psycopg2... line indented 4 spaces.
However, the import continues past those errors. For example:
Yes, that's a report that the one message with Message-ID: <04b901d993ca$d6cf3180$846d9480$@gmail.com> failed to import, but it doesn't affect the rest of the import.
It all eventually completes with exit code 0. Generally, successful. So it's unclear if a skipped message is the cause.
It isn't, but the "psycopg2.OperationalError: SSL connection has been closed unexpectedly" message is curious. I would find that message in the mbox file and try to run the import again with a mbox containing just the one message with a --since option early enough to not skip it and also the --no-sync-mailman option to avoid that time consuming process which only will update HyperKitty's Sender record for this sender with Mailman core's user_id if it hasn't already been done for this sender.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
Thanks for the reply, I may have questions about those debugging suggestions, but it's late in the week, will continue next week. That will also give cron a few more chances to run in the meantime. Since you mentioned which job it is, maybe I will also run that manually to see what the results are.
Hi,
Ok, there's more information. :-)
- "Threads by Month" is populated with messages from the last month.
- "All Threads" is populated with messages from the last month. "Recently Active Discussions" (the main page) is empty.
After 3 days of cron jobs running, no changes. Then I manually ran the recent_threads_cache update:
sudo -u mailman3-web MAILMAN_WEB_CONFIG=/var/lib/mailman3/web/project/settings.py /opt/mailman3/bin/mailman-web runjob -v 3 hyperkitty recent_threads_cache
It completes successfully, but didn't output much. Still "Recently Active Discussions" is empty.
Continuing on, Hyperkitty is running as a gunicorn process via systemd. That is not out-of-the-ordinary.
[Service] ExecStart=/opt/mailman3/bin/gunicorn ...
Try this:
systemctl reload mailman3-web or systemctl restart mailman3-web
The "Recently Active Discussions" listing is refreshed! That fixes it.
Related issue, #2
On the "Recently Active Discussions" page, click on a message, a test message, and then "Delete this thread". It's deleted.
Now "Recently Active Discussions" hangs. A spinning icon, indefinitely. In the logs file: "hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist." Re-running recent_threads_cache, and restarting Django, fixes it. However in production you wouldn't want to wait 24 hrs, and also manually restart the web.
Would it be helpful if I opened gitlab issues for the above items? ( or, these are expected issues, my configuration is wrong )
On 6/3/24 8:55 AM, Sam Darwin wrote:
Try this:
systemctl reload mailman3-web or systemctl restart mailman3-web
The "Recently Active Discussions" listing is refreshed! That fixes it.
Related issue, #2
On the "Recently Active Discussions" page, click on a message, a test message, and then "Delete this thread". It's deleted.
Now "Recently Active Discussions" hangs. A spinning icon, indefinitely. In the logs file: "hyperkitty.models.thread.Thread.DoesNotExist: Thread matching query does not exist." Re-running recent_threads_cache, and restarting Django, fixes it. However in production you wouldn't want to wait 24 hrs, and also manually restart the web.
As you say, these issues may be related. The latter is already reported at https://gitlab.com/mailman/hyperkitty/-/issues/504 and as it says there, I am unable to duplicate it; it may be related to which database manager you use.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
I can provide access to a test server with identical configuration it this can be of any help
participants (3)
-
Mark Sapiro -
Roberto Puzzanghera -
Sam Darwin