Changes to 'refs/tags/1.3.5'
by Matt Domsch
Tag '1.3.5' created by Matt Domsch <matt(a)domsch.com> at 2010-05-27 01:32 +0000
v1.3.5
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
iQEVAwUAS/3Lnlq36sXWI9AEAQjtPggAo7VpkGyxrrR39k6VfAYkJkg8k65Wc/uS
G6asRUed/fLEu9vUY9A28TP+LipzLAhQEsHk0I0bMoi+0FD4lDUEH9GfwoOh4JDM
DTPp4p7C+lspVYEOjcnt5J9tuiphIfQk1c4gmLYjbgT0uBzTZJaPpcnppRrLRXrR
Rh5k8cufw8Ur0DyqA6BOjOAF+oi5A4WLtw1Mi3gNhnET4m8OJkZkSGjYB4GLIsq6
8/vFHVluuql/QlAE09oEkrrqwzmecB6cu84ZGgd8fnSoFwHR5i8T8jarj3HBM13l
3CUCk5aAelE0ntmDpeU+Ps+Af/I/wbcH8WjFlvwG1o2VmSaaXBGV0A==
=yapm
-----END PGP SIGNATURE-----
Changes since v1.3.4:
Adrian Reber (1):
mirrorlist_server: handle bad client IP address input
Matt Domsch (20):
spec: require python-basemap-data too
makefile: add git-sign-push target to remind me
crawler: implement canary mode
crawler: crawl private i2 hosts that also say i2_clients are ok
crawler: send email when marking a whole host not up2date
mirrorlist_server: ignore SIGHUP in child - it'll exit soon enough
mirrorlist_server: increase socket request queue length to 300 (still limited by kernel options to 128)
mirrorlist_server: set request_queue_size earlier
mirrorlist_server: don't ignore SIGCHLD
mirrorlist tester: more printing, run forever
mirrorlist_client: sanitize input into UTF-8
mirrorlist_client: add 60sec timeout to reading from the server
mirrorlist_client: use select() waiting on the response from mirrorlist_server
mirrorlist_client: a better way to handle socket timeouts
mirrorlist_client: leave query params as utf8
mirrorlist_server: handle more failures gracefully
mirrorlist_client: add mirrorlist_client.noreverseproxy config parameter
umdl: don't use move_repository_from_development()
bump version
Makefile: fix git-sign-push
---
Makefile | 9 ++-
TODO | 2
mirrorlist-server/apache/mirrorlist-server.conf | 4 +
mirrorlist-server/mirrorlist_client.wsgi | 44 +++++++++++----
mirrorlist-server/mirrorlist_server.py | 29 +++++++---
mirrorlist-server/test/server_tester.py | 29 +++++++---
mirrormanager.spec.in | 25 ++++++++
server/crawler | 7 +-
server/crawler_perhost | 68 +++++++++++++++++++++---
server/dev.cfg | 5 +
server/update-master-directory-list | 21 -------
11 files changed, 180 insertions(+), 63 deletions(-)
---
13 years, 11 months
2 commits - Makefile mirrormanager.spec.in
by Matt Domsch
Makefile | 6 +++---
mirrormanager.spec.in | 23 +++++++++++++++++++++++
2 files changed, 26 insertions(+), 3 deletions(-)
New commits:
commit 9ba5bbb5291c74b94d556bc8e88bedc7ed89712f
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Wed May 26 20:32:00 2010 -0500
Makefile: fix git-sign-push
diff --git a/Makefile b/Makefile
index 4beaf82..d791154 100644
--- a/Makefile
+++ b/Makefile
@@ -122,6 +122,6 @@ publish: sign
scp dist/* fedorahosted.org:/srv/web/releases/m/i/mirrormanager/
git-sign-push:
- git tag -s v$(RELEASE_VERSION) $(RELEASE_VERSION)
+ git tag -s -m v$(RELEASE_VERSION) $(RELEASE_VERSION)
git push
git push --tags
commit f4f1a1693b6a88ab0d871399d18e10ae78f0a521
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Wed May 26 20:30:36 2010 -0500
bump version
diff --git a/Makefile b/Makefile
index 9470d8c..4beaf82 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
-RELEASE_DATE := "26-Apr-2010"
+RELEASE_DATE := "26-May-2010"
RELEASE_MAJOR := 1
RELEASE_MINOR := 3
-RELEASE_EXTRALEVEL := .4
+RELEASE_EXTRALEVEL := .5
RELEASE_NAME := mirrormanager
RELEASE_VERSION := $(RELEASE_MAJOR).$(RELEASE_MINOR)$(RELEASE_EXTRALEVEL)
RELEASE_STRING := $(RELEASE_NAME)-$(RELEASE_VERSION)
diff --git a/mirrormanager.spec.in b/mirrormanager.spec.in
index 6a41d09..1c8e00f 100644
--- a/mirrormanager.spec.in
+++ b/mirrormanager.spec.in
@@ -85,6 +85,29 @@ exit 0
%changelog
+* Wed May 26 2010 Matt Domsch <mdomsch(a)fedoraproject.org> - 1.3.5-1
+- Adrian Reber (1):
+ mirrorlist_server: handle bad client IP address input
+- Matt Domsch (18):
+ spec: require python-basemap-data too
+ makefile: add git-sign-push target to remind me
+ crawler: implement canary mode
+ crawler: crawl private i2 hosts that also say i2_clients are ok
+ crawler: send email when marking a whole host not up2date
+ mirrorlist_server: ignore SIGHUP in child - it'll exit soon enough
+ mirrorlist_server: increase socket request queue length to 300 (still limited by kernel options to 128)
+ mirrorlist_server: set request_queue_size earlier
+ mirrorlist_server: don't ignore SIGCHLD
+ mirrorlist tester: more printing, run forever
+ mirrorlist_client: sanitize input into UTF-8
+ mirrorlist_client: add 60sec timeout to reading from the server
+ mirrorlist_client: use select() waiting on the response from mirrorlist_server
+ mirrorlist_client: a better way to handle socket timeouts
+ mirrorlist_client: leave query params as utf8
+ mirrorlist_server: handle more failures gracefully
+ mirrorlist_client: add mirrorlist_client.noreverseproxy config parameter
+ umdl: don't use move_repository_from_development()
+
* Mon Apr 26 2010 Matt Domsch <mdomsch(a)fedoraproject.org> - 1.3.4-1
- Adrian Reber (2):
umdl: fix failure when updating from rsync listings.
13 years, 11 months
server/update-master-directory-list
by Matt Domsch
server/update-master-directory-list | 21 +--------------------
1 file changed, 1 insertion(+), 20 deletions(-)
New commits:
commit bd0fe39e545683463af5897cafd1c83d9abca55a
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Wed May 26 20:26:22 2010 -0500
umdl: don't use move_repository_from_development()
For Fedora 13, when releases/13 was created, this routine moved the
Repository pointers from development/13 to releases/13 as expected.
However, releases/13 was not world readable yet, so all users of
mirrorlist?repo=fedora-13 broke.
I'm not sure we can do this in umdl. We would have to know the
permissions, and hope they don't get more restrictive after creation.
Ugh. So, nuke it from umdl for now, and consider doing this in the
releaseday script (to be added at some point).
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 9de759a..4bdc226 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -195,18 +195,6 @@ def make_repomd_file_details(dir):
fd = FileDetail(directory=dir, filename='repomd.xml', sha1=sha1, md5=md5, sha256=sha256, sha512=sha512,
timestamp=timestamp, size=size)
-
-def move_repository_from_development(prefix, arch, newdir):
- try:
- repo = Repository.selectBy(prefix=prefix, arch=arch)[0]
- except:
- return None
-
- if repo.directory is not None and newdir.name is not None:
- if '/development' in repo.directory.name and '/releases' in newdir.name:
- repo.directory = newdir
- return repo
-
def make_repository(dir, category):
repo = None
path = dir.name[len(category.topdir.name)+1:]
@@ -223,17 +211,10 @@ def make_repository(dir, category):
try:
repo = Repository(name=name, category=category, version=ver, arch=arch, directory=dir, prefix=prefix)
except:
- # repo exists, but it may be one we want to move
- try:
- repo = move_repository_from_development(prefix, arch, dir)
- except:
- pass
+ pass
return repo
-
-
-
def nuke_gone_directories(category, category_directories):
""" deleting a Directory has a ripple effect through the whole
database. Be really sure you're ready do to this. It comes
13 years, 11 months
mirrorlist-server/apache mirrorlist-server/mirrorlist_client.wsgi
by Matt Domsch
mirrorlist-server/apache/mirrorlist-server.conf | 4 ++++
mirrorlist-server/mirrorlist_client.wsgi | 13 ++++++++++---
2 files changed, 14 insertions(+), 3 deletions(-)
New commits:
commit 3b117a9b6059ae0c8449f1cb920d4e52d3ed30e7
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon May 17 16:30:43 2010 -0500
mirrorlist_client: add mirrorlist_client.noreverseproxy config parameter
noted by Adrian Reber, the X-Forwarded-For trick only is useful if you
are running a Reverse Proxy (HTTP Accelerator), such as Fedora
Infrastructure does. If that is not your setup (such as rpmfusion),
then you don't want to look at X-Forwarded-For at all.
diff --git a/mirrorlist-server/apache/mirrorlist-server.conf b/mirrorlist-server/apache/mirrorlist-server.conf
index a26e4b3..87678e0 100644
--- a/mirrorlist-server/apache/mirrorlist-server.conf
+++ b/mirrorlist-server/apache/mirrorlist-server.conf
@@ -12,6 +12,10 @@ WSGIDaemonProcess mirrorlist user=apache processes=6 threads=25 display-name=mir
WSGIScriptAlias /metalink /usr/share/mirrormanager/mirrorlist-server/mirrorlist_client.wsgi
WSGIScriptAlias /mirrorlist /usr/share/mirrormanager/mirrorlist-server/mirrorlist_client.wsgi
+# Set this if you do not have a Reverse Proxy (HTTP Accelerator) that
+# is in front of your application server running this code.
+# SetEnv mirrorlist_client.noreverseproxy 1
+
<Location /mirrorlist>
WSGIProcessGroup mirrorlist
Order deny,allow
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index 36077a1..b009d3b 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -4,6 +4,13 @@
# by Matt Domsch <Matt_Domsch(a)dell.com>
# Licensed under the MIT/X11 license
+# Environment Variables setable via Apache SetEnv directive:
+# mirrorlist_client.noreverseproxy
+# if set (to anything), do not look at X-Forwarded-For headers. This
+# is used in environments that do not have a Reverse Proxy (HTTP
+# accelerator) in front of the application server running this WSGI,
+# to avoid looking "behind" the real client's own forward HTTP proxy.
+
import socket, select
import cPickle as pickle
from string import zfill, atoi, strip, replace
@@ -62,7 +69,7 @@ def real_client_ip(xforwardedfor):
any way."""
return xforwardedfor.split(',')[-1].strip()
-def request_setup(request):
+def request_setup(environ, request):
fields = ['repo', 'arch', 'country', 'path', 'netblock']
d = {}
request_data = request.GET
@@ -75,7 +82,7 @@ def request_setup(request):
if 'ip' in request_data:
client_ip = strip(request_data['ip'])
- elif 'X-Forwarded-For' in request.headers:
+ elif 'X-Forwarded-For' in request.headers and 'mirrorlist_client.noreverseproxy' not in environ:
client_ip = real_client_ip(strip(request.headers['X-Forwarded-For']))
else:
client_ip = request.environ['REMOTE_ADDR']
@@ -122,7 +129,7 @@ def application(environ, start_response):
request = WSGIRequest(environ)
response = WSGIResponse()
- d = request_setup(request)
+ d = request_setup(environ, request)
try:
r = get_mirrorlist(d)
13 years, 11 months
mirrorlist-server/mirrorlist_server.py
by Matt Domsch
mirrorlist-server/mirrorlist_server.py | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
New commits:
commit bb8c3038cf9f76fa39e8580a990d7d9edef7f43c
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon May 17 10:15:34 2010 -0500
mirrorlist_server: handle more failures gracefully
Handle being passed a bad country= value.
Catch all failures in do_mirrorlist() and return a proper error
message. This should also prevent failures in mirrorlist_server from
causing mirrorlist_client() to spin indefinitely waiting for an answer.
diff --git a/mirrorlist-server/mirrorlist_server.py b/mirrorlist-server/mirrorlist_server.py
index e909927..0d73469 100755
--- a/mirrorlist-server/mirrorlist_server.py
+++ b/mirrorlist-server/mirrorlist_server.py
@@ -262,7 +262,7 @@ def do_countrylist(kwargs, cache, clientCountry, requested_countries, header):
def get_same_continent_countries(clientCountry, requested_countries):
result = []
for r in requested_countries:
- if r is not None:
+ if r in country_continents:
requestedCountries = [c.upper() for c in continents[country_continents[r]] \
if c != clientCountry ]
result.extend(requestedCountries)
@@ -690,17 +690,22 @@ class MirrorlistHandler(StreamRequestHandler):
pass
try:
- r = do_mirrorlist(d)
+ try:
+ r = do_mirrorlist(d)
+ except:
+ raise
message = r['message']
results = r['results']
resulttype = r['resulttype']
returncode = r['returncode']
except:
- message='# Server Error'
+ message=u'# Bad Request'
+ returncode = 400
+ results = []
+ resulttype = 'mirrorlist'
if d['metalink']:
resulttype = 'metalink'
- results = errordoc(d['metalink'], message)
- returncode = 500
+ results = errordoc(d['metalink'], message)
del d
del p
13 years, 11 months
mirrorlist-server/mirrorlist_server.py
by Matt Domsch
mirrorlist-server/mirrorlist_server.py | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
New commits:
commit 2530807e1216786ea5dac8110cbb27a2f25103a3
Author: Adrian Reber <adrian(a)lisas.de>
Date: Sun May 16 17:15:22 2010 -0500
mirrorlist_server: handle bad client IP address input
Signed-off-by: Matt Domsch <matt(a)domsch.com>
diff --git a/mirrorlist-server/mirrorlist_server.py b/mirrorlist-server/mirrorlist_server.py
index 0de7132..e909927 100755
--- a/mirrorlist-server/mirrorlist_server.py
+++ b/mirrorlist-server/mirrorlist_server.py
@@ -298,7 +298,10 @@ def do_internet2(kwargs, cache, clientCountry, header):
client_ip = kwargs['client_ip']
if client_ip == 'unknown':
return (header, hostresults)
- ip = IP(client_ip)
+ try:
+ ip = IP(client_ip)
+ except:
+ return (header, hostresults)
asn = lookup_ip_asn(internet2_tree, ip)
if asn is not None:
header += 'Using Internet2 '
@@ -312,7 +315,10 @@ def do_asn(kwargs, cache, header):
client_ip = kwargs['client_ip']
if client_ip == 'unknown':
return (header, hostresults)
- ip = IP(client_ip)
+ try:
+ ip = IP(client_ip)
+ except:
+ return (header, hostresults)
asn = lookup_ip_asn(global_tree, ip)
if asn is not None and asn in asn_host_cache:
for hostid in asn_host_cache[asn]:
13 years, 11 months
mirrorlist-server/mirrorlist_client.wsgi
by Matt Domsch
mirrorlist-server/mirrorlist_client.wsgi | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
New commits:
commit 9251a20e8ff8239bae2c74e64ad20a4645423ae9
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Wed May 12 22:59:46 2010 -0500
mirrorlist_client: leave query params as utf8
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index 3508f19..36077a1 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -93,7 +93,7 @@ def request_setup(request):
for k, v in d.iteritems():
try:
- d[k] = unicode(v, 'utf8', 'ignore').encode('utf8')
+ d[k] = unicode(v, 'utf8', 'replace')
except:
pass
return d
13 years, 11 months
mirrorlist-server/mirrorlist_client.wsgi
by Matt Domsch
mirrorlist-server/mirrorlist_client.wsgi | 17 ++++++++---------
1 file changed, 8 insertions(+), 9 deletions(-)
New commits:
commit 1aa19dfed950c209ad5a2ddf48e2b828b50c07ee
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Wed May 12 13:52:51 2010 -0500
mirrorlist_client: a better way to handle socket timeouts
blocking sockets, calling recv(), may block forever if the server end
doesn't send anything for some reason. Don't let that happen. Python
has a socket.settimeout() capability. We'll use that to let any
individual operation (except the select()) take up to 5 seconds (they
should all be in the microsecond range, so this is very generous), and
let select() continue to wait for 60 seconds for the server to respond
at all.
If a timeout happens, an exception is raised, which is caught by the
caller and a HTTP 503 returned to the web client.
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index 15b3a15..3508f19 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -13,14 +13,14 @@ import cStringIO
from datetime import datetime, timedelta
socketfile = '/var/run/mirrormanager/mirrorlist_server.sock'
-request_timeout = 60 # seconds
+select_timeout = 60 # seconds
+timeout = 5 # seconds
def get_mirrorlist(d):
- try:
- s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
- s.connect(socketfile)
- except:
- raise
+ # any exceptions or timeouts raised here get handled by the caller
+ s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ s.settimeout(timeout)
+ s.connect(socketfile)
p = pickle.dumps(d)
del d
@@ -33,8 +33,7 @@ def get_mirrorlist(d):
del p
# wait for other end to start writing
- expiry = datetime.utcnow() + timedelta(seconds=request_timeout)
- rlist, wlist, xlist = select.select([s],[],[],request_timeout)
+ rlist, wlist, xlist = select.select([s],[],[],select_timeout)
if len(rlist) == 0:
s.shutdown(socket.SHUT_RD)
raise socket.timeout
@@ -48,7 +47,7 @@ def get_mirrorlist(d):
readlen = 0
p = ''
- while readlen < resultsize and datetime.utcnow() < expiry:
+ while readlen < resultsize:
p += s.recv(resultsize - readlen)
readlen = len(p)
results = pickle.loads(p)
13 years, 11 months
mirrorlist-server/mirrorlist_client.wsgi
by Matt Domsch
mirrorlist-server/mirrorlist_client.wsgi | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
New commits:
commit 7cd05b296ab426c386e99c3ff6f7143fbf6ed052
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Wed May 12 08:59:16 2010 -0500
mirrorlist_client: use select() waiting on the response from mirrorlist_server
Client was spinning waiting for read() to complete, during the time
the server was doing its thinking. Instead, use select() to sleep
until the server has data to return. This should reduce CPU time
spent in the client considerably.
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index cc4416c..15b3a15 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -4,7 +4,7 @@
# by Matt Domsch <Matt_Domsch(a)dell.com>
# Licensed under the MIT/X11 license
-import socket
+import socket, select
import cPickle as pickle
from string import zfill, atoi, strip, replace
from paste.wsgiwrappers import *
@@ -32,24 +32,29 @@ def get_mirrorlist(d):
s.shutdown(socket.SHUT_WR)
del p
+ # wait for other end to start writing
+ expiry = datetime.utcnow() + timedelta(seconds=request_timeout)
+ rlist, wlist, xlist = select.select([s],[],[],request_timeout)
+ if len(rlist) == 0:
+ s.shutdown(socket.SHUT_RD)
+ raise socket.timeout
+
readlen = 0
resultsize = ''
while readlen < 10:
resultsize += s.recv(10 - readlen)
readlen = len(resultsize)
resultsize = atoi(resultsize)
-
- expiry = datetime.utcnow() + timedelta(seconds=request_timeout)
+
readlen = 0
p = ''
while readlen < resultsize and datetime.utcnow() < expiry:
p += s.recv(resultsize - readlen)
readlen = len(p)
-
- s.shutdown(socket.SHUT_RD)
results = pickle.loads(p)
del p
+ s.shutdown(socket.SHUT_RD)
return results
def real_client_ip(xforwardedfor):
13 years, 11 months
mirrorlist-server/mirrorlist_client.wsgi
by Matt Domsch
mirrorlist-server/mirrorlist_client.wsgi | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
New commits:
commit 4976394a2f988843baf1d3e490dcc5dd3e74b1ea
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Tue May 11 23:14:15 2010 -0500
mirrorlist_client: add 60sec timeout to reading from the server
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index dd6d1d9..cc4416c 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -10,8 +10,10 @@ from string import zfill, atoi, strip, replace
from paste.wsgiwrappers import *
import gzip
import cStringIO
+from datetime import datetime, timedelta
socketfile = '/var/run/mirrormanager/mirrorlist_server.sock'
+request_timeout = 60 # seconds
def get_mirrorlist(d):
try:
@@ -37,9 +39,10 @@ def get_mirrorlist(d):
readlen = len(resultsize)
resultsize = atoi(resultsize)
+ expiry = datetime.utcnow() + timedelta(seconds=request_timeout)
readlen = 0
p = ''
- while readlen < resultsize:
+ while readlen < resultsize and datetime.utcnow() < expiry:
p += s.recv(resultsize - readlen)
readlen = len(p)
13 years, 11 months