Repository :
http://git.fedorahosted.org/cgit/
On branch : master
---------------------------------------------------------------
commit 9becec2ffa53bafb1f791803c9527378af95dcd6
Author: Matt Domsch <matt(a)domsch.com>
Date: Sat Jul 27 12:22:04 2013 -0500
crawler_perhost: limit try_later_delay to 4 seconds
---------------------------------------------------------------
server/crawler_perhost | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/crawler_perhost b/server/crawler_perhost
index 2f72a60..dfc9292 100755
--- a/server/crawler_perhost
+++ b/server/crawler_perhost
@@ -774,7 +774,7 @@ def per_host(host, options):
logger.warning("Host %s (id=%d) does not have HTTP Keep-Alives
enabled." % (host.name, host.id))
time.sleep(try_later_delay)
- if try_later_delay < 64:
+ if try_later_delay < 8:
try_later_delay = try_later_delay << 1
except: