Bernie,
Thanks for the heads-up!
Some sites may prefer a /robots.txt with
User-agent: * Disallow: /mailman3/ Disallow: /archives/ Disallow: /accounts/ Disallow: /user-profile/
and if these paths are enabled
Disallow: /postorius/ Disallow: /hyperkitty/ Disallow: /admin/
Note that the latter disallows indexing of /admin/docs/ if that URL is enabled. Presumably that's not a problem, since the general public should be going to djangoproject.com for documentation anyway.
Bernie Hoeneisen writes:
Not sure though, how they harvested the URL that is triggering the error. Maybe that URL is exposed somewhere, e.g. in Postorius or Hyperkitty.
I would imagine that somebody naive (eg, no robots.txt at all) has the top page with a sign-in or sign-up button exposed, which will lead to a page with social login buttons, including the fedora/login button.
-- GNU Mailman consultant (installation, migration, customization) Sirius Open Source https://www.siriusopensource.com/ Software systems consulting in Europe, North America, and Japan