On Mon, 26 Jan 2015 07:53:27 -0700 Brandon Vincent
<Brandon.Vincent(a)asu.edu> wrote:
> On Mon, Jan 26, 2015 at 7:21 AM, Florian Weimer <fweimer(a)redhat.com>
> wrote:
>> Any idea why?
>
>
https://lists.fedoraproject.org/robots.txt
>
> User-agent: * Crawl-delay: 10
>
> From Bing, "This means the higher your crawl delay is, the fewer pages
> BingBot will crawl. As crawling fewer pages may result in getting less
> content indexed, we usually do not recommend it, although we also
> understand that different web sites may have different bandwidth
> constraints." [1].
>
> [1]
>
http://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-...
>
Not sure that explains why there are no results at all though.
I think we added the Crawl-delay several years ago when we were having
storage issues. We could definitely try removing it and see if things
improve.
Yes please. We definitely should get archives indexed by search engines!
--
Petr Spacek @ Red Hat