List enumeration problem
On our mailman instance we see the following enumeration of ALL lists and ALL users consistently every 7.5 minutes. Does anyone know why this would be the case? We have a lot of mailing lists so it slows our machine down a lot.
Apr 30 15:31:36 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/a@example.com HTTP/1.1" 200 310 Apr 30 15:31:36 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/a@example.com/config HTTP/1.1" 200 2021 Apr 30 15:31:36 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/b@example.com HTTP/1.1" 200 325 Apr 30 15:31:37 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/b@example.com/config HTTP/1.1" 200 2108 Apr 30 15:31:37 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/c@example.com HTTP/1.1" 200 326 Apr 30 15:31:37 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/c@example.com/config HTTP/1.1" 200 2049 Apr 30 15:31:37 2018 (39026) 127.0.0.1 - - "GET /3.0/lists/d@example.com HTTP/1.1" 200 324 ...
Apr 30 16:26:28 2018 (9581) 127.0.0.1 - - "GET /3.0/users/a@a HTTP/1.1" 404 13 Apr 30 16:26:28 2018 (9581) 127.0.0.1 - - "GET /3.0/users/b@b HTTP/1.1" 404 13 Apr 30 16:26:28 2018 (9581) 127.0.0.1 - - "GET /3.0/users/c@c HTTP/1.1" 404 13 Apr 30 16:26:28 2018 (9581) 127.0.0.1 - - "GET /3.0/users/d@d HTTP/1.1" 404 13
This appears to be logs from the backend. Do you have corresponding entries from the frontend?
I have a couple of explanations for this:
Normal operation: Most rest calls don't support batch processing, so the frontend needs to issue multiple calls.
Synchronization between frontend and backend. Some stuff is synced between the two, it might be that...
Hyperkitty cron jobs Hyperkitty has a couple of cronjobs to do maintenance on the database and such.
If you have logs from the frontend, we can know for sure what's causing this. 1 is a known issue and is going to get fixed eventually... (especially since very large Mailman instances are popping up...)
Simon,
The thing is that there doesn't appear to be any corresponding entries in the front end logs. There isn't anyone hitting postorius for the lists that are being enumerated.
About the three items you mentione:
I hope that this can be fixed soon! There are several pages on our mailman instance that we had to write a separate back-end API to query the data, because a single user hitting a single page in postorius was causing a denial of service.
What sort of stuff is synced between the front and backends? Any chance you can point me to somewhere in the code where this is happening? We may need to rewrite something to streamline this as well, but it would be nice to know where we need to go poking around.
We have some hyperkitty cron jobs set up, but at the moment they have been scaled back somewhat. I do know that when this enumeration is going on our hyperkitty jobs aren't being run.
We (I work with Kaleb) are one of those very large Mailman instances you mention, so yep, we're seeing what you're talking about! I think in our case the most likely root cause is your #2 above, so any guidance you can shoot in our direction would be much appreciated.
Thanks!
-Darren
On Fri, May 11, 2018 at 3:53 AM, Simon Hanna <simon.h4nn4@gmail.com> wrote:
This appears to be logs from the backend. Do you have corresponding entries from the frontend?
I have a couple of explanations for this:
Normal operation: Most rest calls don't support batch processing, so the frontend needs to issue multiple calls.
Synchronization between frontend and backend. Some stuff is synced between the two, it might be that...
Hyperkitty cron jobs Hyperkitty has a couple of cronjobs to do maintenance on the database and such.
If you have logs from the frontend, we can know for sure what's causing this. 1 is a known issue and is going to get fixed eventually... (especially since very large Mailman instances are popping up...)
Mailman-users mailing list mailman-users@mailman3.org https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
- What sort of stuff is synced between the front and backends? Any chance you can point me to somewhere in the code where this is happening? We may need to rewrite something to streamline this as well, but it would be nice to know where we need to go poking around. That would be django-mailman3 specifically signals.py and lib/mailman.py
I didn't write any of that code, I just know it's there. I'm not even sure when it's triggered... :-)
participants (4)
-
Darren Smith -
kawhite＠ancestry.com -
Simon Hanna -
Simon Hanna