On Thu, Apr 19, 2018 at 08:54:21AM +0200, Adrian Reber wrote:
As discussed previously I would like to change the crawler to crawl
category separately. The goal is to reduce the load on the database by
distributing the crawling better over the whole day and to reduce the
chance of mirrors being disabled because of the high database load.
This should also remove the need for mirror administrators to create
multiple hosts in MirrorManager to work around the 4 hours timeout per
On the mirror-admin mailing list there have been reports that mirrors
are no longer part of the mirrorlist/metalink.
With the switch to category based crawling a bug in the crawler was
It is fixed and the crawler should now mark directories correctly during