For some reason, yahoo's bot has been hitting our taskotron instance
really hard all of a sudden - slowing it down enough to trigger nagios
alerts twice this week.
There is a robots.txt served by the buildmaster instance but since that
isn't at the webroot in our setup, bots may not read it.
I'd like to add a robots.txt to the public facing taskotron instances
so that we can avoid this in the future. Rendered content of the
robots.txt is at the end of this email.
I've already deployed the file to taskotron-dev which can be seen at:
http://taskotron-dev.fedoraproject.org/robots.txt
Proposed content would be at:
http://taskotron.stg.fedoraproject.org/robots.txt
http://taskotron.fedoraproject.org/robots.txt
Tim
User-agent: *
Disallow: /taskmaster/waterfall
Disallow: /taskmaster/builders
Disallow: /taskmaster/changes
Disallow: /taskmaster/buildslaves
Disallow: /taskmaster/schedulers
Disallow: /taskmaster/one_line_per_build
Disallow: /taskmaster/grid
Disallow: /taskmaster/tgrid
Disallow: /taskmaster/json