Fix, kind of, of 'Internal Server Error'
I was getting several thousand error messages a day complaining about this. I would also find that the web interface would time-out/die - going away to make a cup of tea was the only things to do -- ie just wait.
I am getting far fewer of these today after I tweaked by robots.txt. The problem was search engines (and maybe AI monsters) crawling the site and over loading it. The machine has 1.5GB of RAM & plenty of fast (SSD) swap. It copes well enough except when the crawlers come in.
So, 2 fixes:
Crawl-delay: 10
Do not visit more frequently than every 10 seconds. Google ignores this as it expects you to waste time on their search console.
Disallow: /mailman3/
Do not visit at all. However if you do this then archives will not be indexed.
All that I can say is: works for me, I have 1% of the errors that I had previously.
-- Alain Williams Linux/GNU Consultant - Mail systems, Web sites, Networking, Programmer, IT Lecturer. +44 (0) 787 668 0256 https://www.phcomp.co.uk/ Parliament Hill Computers. Registration Information: https://www.phcomp.co.uk/Contact.html #include <std_disclaimer.h>
participants (1)
-
Alain D D Williams