[mirrormanager] v1.4: model.py: fix user_sites() variable name (80dd827)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 80dd82765f337e7f9a409b4e351c848bdb5aca59
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Fri Jun 21 07:13:39 2013 -0500
model.py: fix user_sites() variable name
>---------------------------------------------------------------
server/mirrormanager/model.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/mirrormanager/model.py b/server/mirrormanager/model.py
index bba2ee7..6ebadc7 100644
--- a/server/mirrormanager/model.py
+++ b/server/mirrormanager/model.py
@@ -152,7 +152,7 @@ def user_sites(identity):
SiteAdmin.q.username == identity.current.user_name)))
for site in query_result:
for h in site.hosts:
- result.append((site.id, site.name, host.id, host.name))
+ result.append((site.id, site.name, h.id, h.name))
sites = _min_sites_and_hosts(result)
return sites
10 years, 12 months
[mirrormanager] v1.4: add count-mirrorlist-hits script (f7332fe)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit f7332fe849fa0f80e977936c8d83d91d11e25b55
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Thu Jun 20 23:15:02 2013 -0500
add count-mirrorlist-hits script
>---------------------------------------------------------------
server/count-mirrorlist-hits | 63 ++++++++++++++++++++++++++++++++++++++++++
1 files changed, 63 insertions(+), 0 deletions(-)
diff --git a/server/count-mirrorlist-hits b/server/count-mirrorlist-hits
new file mode 100755
index 0000000..a815794
--- /dev/null
+++ b/server/count-mirrorlist-hits
@@ -0,0 +1,63 @@
+#!/usr/bin/python
+import sys
+import optparse
+
+
+stats = dict(I2 = 0,
+ netblocks = 0,
+ asn = 0,
+ geoip = 0,
+ country = 0,
+ continent = 0)
+stats['global'] = 0
+stats['None'] = 0
+last_zone = 'geoip'
+
+
+def handle_repeated(line):
+ return int(line[-2])
+
+def parse_line(line):
+ global stats
+ global last_zone
+ line = line.split()
+ if line[4] != 'mirrorlist:': return
+ if line[-1] == 'times':
+ additional = handle_repeated(line)
+ stats[last_zone] += additional
+ return
+
+ where = line[-1]
+ last_zone = where
+ stats[where] += 1
+
+def total():
+ total = 0;
+ for v in stats.values():
+ total += v
+ return total
+
+def read_files(filenames):
+ for fname in filenames:
+ f = open(fname)
+ while True:
+ line = f.readline()
+ if not line: break
+ parse_line(line)
+ f.close()
+
+
+
+def main():
+ parser = optparse.OptionParser(usage=sys.argv[0] + " [options]")
+ (options, args) = parser.parse_args()
+ read_files(args)
+ t = total()
+ print "total hits: %s" % t
+ for k, v in stats.iteritems():
+ print "%s had %s hits, %s%%" % (k, v, v*100/t)
+ return 0
+
+if __name__ == "__main__":
+ rc = main()
+ sys.exit(rc)
11 years
[mirrormanager] v1.4: model.py: fix sqlbuilder imports (af5dcda)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit af5dcda8cda236bbb34c0317168ca356656e5da0
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Thu Jun 20 17:13:06 2013 -0500
model.py: fix sqlbuilder imports
>---------------------------------------------------------------
server/mirrormanager/model.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/mirrormanager/model.py b/server/mirrormanager/model.py
index 5ac4a92..bba2ee7 100644
--- a/server/mirrormanager/model.py
+++ b/server/mirrormanager/model.py
@@ -1,6 +1,6 @@
from sqlobject import *
from sqlobject.converters import sqlrepr
-from sqlobject.sqlbuilder import RLIKE
+from sqlobject.sqlbuilder import RLIKE, AND, OR, INNERJOINOn
from turbogears import identity, config
from datetime import datetime
import time
11 years
[mirrormanager] v1.4: set mirrorlist WSGI processes=45, threads=1 (2c99120)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 2c9912075a42707ff381e73ae7a009fb6f679a43
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Thu Jun 20 12:22:29 2013 -0500
set mirrorlist WSGI processes=45, threads=1
We had what appears to be a lockup of mod_wsgi / apache on one of the
Fedora mirrorlist servers overnight. It may be related to this change
in MM 1.4, where we had set it to 6 processes, 25 threads each. It's
worth a try to change it back to what FI has been running in
production since 2010.
>---------------------------------------------------------------
mirrorlist-server/apache/mirrorlist-server.conf | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/mirrorlist-server/apache/mirrorlist-server.conf b/mirrorlist-server/apache/mirrorlist-server.conf
index 6d81bf1..a3ca826 100644
--- a/mirrorlist-server/apache/mirrorlist-server.conf
+++ b/mirrorlist-server/apache/mirrorlist-server.conf
@@ -8,7 +8,7 @@ Alias /static /var/lib/mirrormanager/mirrorlists/static/
ExpiresDefault "modification plus 1 hour"
</Directory>
-WSGIDaemonProcess mirrorlist user=apache processes=6 threads=25 display-name=mirrorlist maximum-requests=1000
+WSGIDaemonProcess mirrorlist user=apache processes=45 threads=1 display-name=mirrorlist maximum-requests=1000
WSGIScriptAlias /metalink /usr/share/mirrormanager/mirrorlist-server/mirrorlist_client.wsgi
WSGIScriptAlias /mirrorlist /usr/share/mirrormanager/mirrorlist-server/mirrorlist_client.wsgi
11 years
[mirrormanager] v1.4: add to TODO (4fee834)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 4fee8347a60ad1ac3cf6ca74c21b0b6ba2fbe1fe
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Tue Jun 18 13:57:56 2013 -0500
add to TODO
>---------------------------------------------------------------
TODO | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/TODO b/TODO
index 28dea8c..ccb3203 100644
--- a/TODO
+++ b/TODO
@@ -10,6 +10,14 @@ Short Term:
scattered throughout. Ugly. Make it easy to point to custom css and
custom theme, so the Fedora theme becomes just separate theme.
+* clean up add-instrepos and mv-instrepos - ideally umdl would handle these.
+* move Fedora-specific apps into a server/fedora directory, add /usr/share/mirrormanager/server to sys.path.
+* improve and use errorcheck code - lots mirrors seem to be missing categories or URLs
+* crawlers failing after a 90min timeout - why?
+* make sure umdl handles SIGHUP, re-opens its log files.
+* make sure crawlers handle SIGHUP, reopens their log files.
+* consider making http redirects only redirect to http servers, not FTP (fixes bfo/ipxe)
+
Long Term:
11 years
[mirrormanager] v1.4: crawler: symlink fix, printing fix (61eb2ed)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 61eb2ed9b5673868b43acd6303fb1565f3c0d784
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon Jun 17 22:48:14 2013 -0500
crawler: symlink fix, printing fix
1) when doing rsync crawling, allow symlinks to have the "wrong" size
for the file.
2) print a line when we change from not up-to-date to up-to-date, and
a different line for vice versa.
>---------------------------------------------------------------
server/crawler_perhost | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/server/crawler_perhost b/server/crawler_perhost
index 2e3e84c..0156c6c 100755
--- a/server/crawler_perhost
+++ b/server/crawler_perhost
@@ -327,6 +327,8 @@ def sync_hcds(host, host_category_dirs):
hcd.sync()
if up2date == False:
print >>logfile, "Directory %s is not up-to-date on this host." % d.name
+ else:
+ print >>logfile, d.name
current_hcds[hcd] = True
@@ -480,7 +482,7 @@ def try_percategory(trydirs, url, host_category_dirs, hc, host, categoryPrefixLe
else:
key = name + "/" + file
try:
- if rsync[key]['size'] != d.files[file]['size']:
+ if rsync[key]['size'] != d.files[file]['size'] and not rsync[key]['mode'].startswith('l'): # ignore symlink size differences
sys.stdout.write('rsync: file size mismatch %s %s != %s\n' % (file, d.files[file]['size'], rsync[key]['size']))
sys.stdout.flush()
all_files = False
@@ -499,7 +501,6 @@ def try_percategory(trydirs, url, host_category_dirs, hc, host, categoryPrefixLe
if all_files is False:
host_category_dirs[(hc, d)] = False
else:
- print d.name
host_category_dirs[(hc, d)] = True
host_category_dirs = add_parents(host_category_dirs, hc, d)
11 years
[mirrormanager] v1.4: fix lib.py imports (9d0a0e8)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 9d0a0e893383782f47ddcafb7e1882897da03df7
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon Jun 17 22:47:54 2013 -0500
fix lib.py imports
>---------------------------------------------------------------
server/mirrormanager/lib.py | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/server/mirrormanager/lib.py b/server/mirrormanager/lib.py
index 7776dde..d94119f 100644
--- a/server/mirrormanager/lib.py
+++ b/server/mirrormanager/lib.py
@@ -1,6 +1,8 @@
import os
import subprocess
+import sys
import tempfile
+import traceback
import types
def createErrorString(tg_errors):
11 years
[mirrormanager] v1.4: fix run_rsync() in umdl and crawler (9b930ab)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 9b930ab244072cd03220f404aeac47304681137b
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon Jun 17 20:46:27 2013 -0500
fix run_rsync() in umdl and crawler
The crawler was failing to properly mark up-to-date or not when using
rsync. It turns out that the rsync results from a fairly complete
mirror can be nearly 100MB itself, which was sitting in memory (and
could cause memory problems). So, let's write that to a safe
temporary file.
Fixing this requires a corresponding change in umdl.
And as tempfile.SpooledTemporaryFile only appeared in python 2.6, and
coindentally that's what RHEL6 has, fix up the spec to require python
2.6 or above.
Furthermore, the crawler was setting all_files outside a loop, instead
of _inside_ the loop where it belonged. That's a bug. And it was
later testing all_files inside a per-file loop, instead of after all
those files had been processed - wasteful.
>---------------------------------------------------------------
mirrormanager.spec.in | 9 +----
server/crawler_perhost | 55 ++++++++++++++++++----------------
server/mirrormanager/lib.py | 30 ++++++++++++-------
server/update-master-directory-list | 4 ++-
4 files changed, 53 insertions(+), 45 deletions(-)
diff --git a/mirrormanager.spec.in b/mirrormanager.spec.in
index 7433642..0faef59 100644
--- a/mirrormanager.spec.in
+++ b/mirrormanager.spec.in
@@ -9,17 +9,12 @@ License: MIT and GPLv2
URL: http://fedorahosted.org/mirrormanager
Source0: https://fedorahosted.org/releases/m/i/%{name}/%{name}-%{version}.tar.xz
BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX)
-BuildRequires: python
+# for tempfile.SpooledTemporaryFile at least. RHEL6 has python 2.6 anyhow.
+BuildRequires: python >= 2.6
Requires: TurboGears, python-IPy, python-GeoIP, wget, yum, psmisc, python-dns, python-fedora >= 0.3.14, python-fedora-turbogears, py-radix, mod_wsgi, python-TurboMail
Requires: supervisor
Requires(pre): shadow-utils
-%define py_ver %(echo `python -c "import sys; print sys.version[:3]"`)
-%if "%{py_ver}" < "2.5"
-# for md5, sha1, sha256, which are native to python 2.5 and higher
-Requires: python-hashlib
-%endif
-
%description
MirrorManager tracks all the content provided on a master mirror
server, and that of all public and private mirrors of that content.
diff --git a/server/crawler_perhost b/server/crawler_perhost
index 0a4a554..2e3e84c 100755
--- a/server/crawler_perhost
+++ b/server/crawler_perhost
@@ -19,6 +19,7 @@ import threading
import urlgrabber
import hashlib
from smtplib import SMTP
+import traceback
from turbogears.database import PackageHub
hostid = None
@@ -151,21 +152,6 @@ class TryLater(Exception): pass
class ForbiddenExpected(Exception): pass
-def get_rsync_dir(url):
- result, output = run_rsync(url, '--no-motd')
-
- if result is False:
- return None
-
- listing = []
-
- for line in output:
- listing.append(line)
-
- output.close()
- return listing
-
-
def get_ftp_dir(hoststate, url, readable, i=0):
if i > 1:
raise TryLater()
@@ -464,18 +450,26 @@ def try_percategory(trydirs, url, host_category_dirs, hc, host, categoryPrefixLe
if not url.endswith('/'):
url += '/'
- listing = get_rsync_dir(url)
+ result, listing = run_rsync(url, '--no-motd')
+ if not result:
+ return None
# put the rsync listing in a dict for easy access
- for line in listing:
+ while True:
+ line = listing.readline()
+ if not line: break
fields = line.split()
- rsync[fields[4]] = {'mode': fields[0], 'size': fields[1], 'date': fields[2], 'time': fields[3]}
+ try:
+ rsync[fields[4]] = {'mode': fields[0], 'size': fields[1], 'date': fields[2], 'time': fields[3]}
+ except IndexError:
+ sys.stderr.write("invalid rsync line: %s\n" % line)
+ sys.stderr.flush()
prefix = ""
- all_files = True
# for all directories in this category
for d in trydirs:
+ all_files = True
# the rsync listing is missing the category part of the url
# remove if from the ones we are comparing it with
name = d.name[categoryPrefixLen:]
@@ -487,21 +481,30 @@ def try_percategory(trydirs, url, host_category_dirs, hc, host, categoryPrefixLe
key = name + "/" + file
try:
if rsync[key]['size'] != d.files[file]['size']:
+ sys.stdout.write('rsync: file size mismatch %s %s != %s\n' % (file, d.files[file]['size'], rsync[key]['size']))
+ sys.stdout.flush()
all_files = False
break
- except:
+ except KeyError: # file is not in the rsync listing
all_files = False
break
+ except Exception, e: # something else went wrong
+ all_files = False
+ exception_msg = "Exception caught in try_percategory()\n"
+ exception_msg += traceback.format_exc(e)
+ sys.stderr.write(exception_msg)
+ sys.stderr.flush()
+ break
- if all_files is False:
- host_category_dirs[(hc, d)] = False
- else:
- host_category_dirs[(hc, d)] = True
- host_category_dirs = add_parents(host_category_dirs, hc, d)
+ if all_files is False:
+ host_category_dirs[(hc, d)] = False
+ else:
+ print d.name
+ host_category_dirs[(hc, d)] = True
+ host_category_dirs = add_parents(host_category_dirs, hc, d)
if len(host_category_dirs) > 0:
sync_hcds(host, host_category_dirs)
- print prefix + " updated"
return True
mark_not_up2date(host, "No host category directories found. Check that your Host Category URLs are correct.")
diff --git a/server/mirrormanager/lib.py b/server/mirrormanager/lib.py
index 1f9db85..7776dde 100644
--- a/server/mirrormanager/lib.py
+++ b/server/mirrormanager/lib.py
@@ -1,5 +1,7 @@
-import types, os
-from subprocess import Popen, PIPE
+import os
+import subprocess
+import tempfile
+import types
def createErrorString(tg_errors):
"""
@@ -89,19 +91,25 @@ def append_value_to_cache(cache, key, value):
return cache
def run_rsync(rsyncpath, extra_rsync_args=None):
- output = None
+ tmpfile = tempfile.SpooledTemporaryFile()
result = True
cmd = "rsync --temp-dir=/tmp -r --exclude=.snapshot --exclude='*.~tmp~'"
if extra_rsync_args is not None:
cmd += ' ' + extra_rsync_args
cmd += ' ' + rsyncpath
try:
- devnull = open('/dev/null', 'r')
- print cmd
- p = Popen(cmd, shell=True, stdin=devnull, stdout=PIPE, bufsize=1, close_fds=True)
- output = p.stdout
- except:
- output = "Unable to run " + args
+ devnull = open('/dev/null', 'rw')
+ sys.stderr.write('invoking %s\n' % cmd)
+ sys.stderr.flush()
+ p = subprocess.Popen(cmd, shell=True, stdin=devnull,
+ stdout=tmpfile, stderr=devnull, close_fds=True, bufsize=-1)
+ p.wait()
+ except Exception, e:
+ msg = "Exception invoking rsync:\n"
+ msg += traceback.format_exc(e)
+ sys.stderr.write(msg+'\n')
+ sys.stderr.flush()
result = False
- devnull.close()
- return (result, output)
+ tmpfile.flush()
+ tmpfile.seek(0)
+ return (result, tmpfile)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 4057532..d380ffc 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -419,7 +419,9 @@ def sync_category_directories(category, category_directories, directory_info):
def parse_rsync_listing(category, f):
topdir_name = category.topdir.name
category_directories = {}
- for line in f:
+ while True:
+ line = f.readline()
+ if not line: break
line.strip()
l = line.split()
if line.startswith('d') and len(l) == 5 and len(l[0]) == 10: # good guess it's a directory line
11 years
[mirrormanager] v1.4: bump release to 1.4.1 (7a0bc8b)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 7a0bc8b1b8350a01a05da896b490f8ab5ab04095
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon Jun 17 13:57:17 2013 -0500
bump release to 1.4.1
>---------------------------------------------------------------
Makefile | 4 ++--
mirrormanager.spec.in | 3 +++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index d2d249c..bc70a7d 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
-RELEASE_DATE := "14-Jun-2013"
+RELEASE_DATE := "17-Jun-2013"
RELEASE_MAJOR := 1
RELEASE_MINOR := 4
-RELEASE_EXTRALEVEL := .0
+RELEASE_EXTRALEVEL := .1
RELEASE_NAME := mirrormanager
RELEASE_VERSION := $(RELEASE_MAJOR).$(RELEASE_MINOR)$(RELEASE_EXTRALEVEL)
RELEASE_STRING := $(RELEASE_NAME)-$(RELEASE_VERSION)
diff --git a/mirrormanager.spec.in b/mirrormanager.spec.in
index 5cfef0a..7433642 100644
--- a/mirrormanager.spec.in
+++ b/mirrormanager.spec.in
@@ -102,6 +102,9 @@ exit 0
%doc LICENSES LICENSE-GPLv2
%changelog
+* Mon Jun 17 2013 Matt Domsch <mdomsch(a)fedoraproject.org> - 1.4.1-1
+- upstream 1.4.1 pure bugfix release
+
* Fri Jun 14 2013 Matt Domsch <mdomsch(a)fedoraproject.org> - 1.4.0-1
- upstream 1.4.0 release
11 years