[mirrormanager] v1.4: sighup supervisord to reopen its log file after rotate (941c0b7)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 941c0b76b3152f3128979e7e2ccce8cf26f812a2
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon Jun 17 07:50:52 2013 -0500
sighup supervisord to reopen its log file after rotate
>---------------------------------------------------------------
server/logrotate.conf | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/server/logrotate.conf b/server/logrotate.conf
index 2fd4b9c..dc9dbe6 100644
--- a/server/logrotate.conf
+++ b/server/logrotate.conf
@@ -4,6 +4,9 @@
daily
dateext
rotate 60
+ postrotate
+ /usr/bin/kill -USR2 $(cat /var/run/supervisord.pid)
+ endscript
}
/var/log/mirrormanager/crawler/*log {
11 years
[mirrormanager] v1.4: end pidfile with a newline (7126cd4)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 7126cd49a799248e5e8a031d4ea1241ce8d09b36
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Mon Jun 17 07:39:47 2013 -0500
end pidfile with a newline
>---------------------------------------------------------------
mirrorlist-server/mirrorlist_server.py | 2 +-
server/mirrormanager/lib.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/mirrorlist-server/mirrorlist_server.py b/mirrorlist-server/mirrorlist_server.py
index 7a73b6a..4bde704 100755
--- a/mirrorlist-server/mirrorlist_server.py
+++ b/mirrorlist-server/mirrorlist_server.py
@@ -968,7 +968,7 @@ def create_pidfile_dir(pidfile):
def write_pidfile(pidfile, pid):
create_pidfile_dir(pidfile)
f = open(pidfile, 'w')
- f.write(str(pid))
+ f.write(str(pid)+'\n')
f.close()
return 0
diff --git a/server/mirrormanager/lib.py b/server/mirrormanager/lib.py
index 0f5fcfa..1f9db85 100644
--- a/server/mirrormanager/lib.py
+++ b/server/mirrormanager/lib.py
@@ -53,7 +53,7 @@ def create_pidfile_dir(pidfile):
def write_pidfile(pidfile, pid):
create_pidfile_dir(pidfile)
f = open(pidfile, 'w')
- f.write(str(pid))
+ f.write(str(pid)+'\n')
f.close()
return 0
11 years
[mirrormanager] v1.4: one more killall removal (a894284)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit a89428489f4f513ea8f777b9f926aa6886165b93
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sun Jun 16 23:03:58 2013 -0500
one more killall removal
>---------------------------------------------------------------
server/update-mirrorlist-server | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/update-mirrorlist-server b/server/update-mirrorlist-server
index 73bc7db..62a4f92 100755
--- a/server/update-mirrorlist-server
+++ b/server/update-mirrorlist-server
@@ -16,6 +16,6 @@ cp -ar ${CACHEDIR}/* ${CACHEDIR}/old/
${MM_DIR}/refresh_mirrorlist_cache -c /etc/mirrormanager/prod.cfg
${MM_DIR}/get_internet2_netblocks ${CACHEDIR}/i2_netblocks.txt
-killall -HUP mirrorlist_server.py
+kill -HUP $(cat /var/run/mirrormanager/mirrorlist_server.pid)
exit 0
11 years
[mirrormanager] v1.4: umdl: fix collision on variable name options (c224b12)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit c224b1229d4e723e14fc7d24fc2caaa138a895c1
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sun Jun 16 20:19:01 2013 -0500
umdl: fix collision on variable name options
>---------------------------------------------------------------
server/update-master-directory-list | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 62d4c46..4057532 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -556,9 +556,9 @@ def main():
setup_arch_version_cache()
for i in config.get('umdl.master_directories'):
try:
- options = i['options']
+ sync_options = i['options']
except KeyError:
- options = None
+ sync_options = None
cname = i['category']
try:
@@ -574,7 +574,7 @@ def main():
continue
if i['type'] == 'rsync':
- sync_directories_using_rsync(i['url'], category, options)
+ sync_directories_using_rsync(i['url'], category, sync_options)
if i['type'] == 'file':
sync_directories_from_file(i['url'], category)
11 years
[mirrormanager] v1.4: Fix very nasty bug where files would be put in the incorrect Directory (742dfc7)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 742dfc7aebd742abd450022cbf7701cb91562f23
Author: Patrick Uiterwijk <puiterwijk(a)gmail.com>
Date: Sun Jun 16 19:44:31 2013 -0500
Fix very nasty bug where files would be put in the incorrect Directory
This was causing the database to have filedetails objects assigned to
the wrong Directory, after which, mirrorlist request would (obviously)
fail.
Sincere thanks to Patrick and Kevin Fenzi for helping track this down
over the weekend.
>---------------------------------------------------------------
server/update-master-directory-list | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 24d5566..62d4c46 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -409,8 +409,8 @@ def sync_category_directories(category, category_directories, directory_info):
# this has to be a second pass to be sure the child repodata/ dir is created in the db first
for dirpath, value in category_directories.iteritems():
d = directory_info[dirpath]
+ D = Directory.get(d['id'])
if value['isRepository']:
- D = Directory.get(d['id'])
make_repository(D, dirpath, category, product_id, topdir)
make_repomd_file_details(dirpath, D)
Directory.ageFileDetails()
11 years
[mirrormanager] v1.4: umdl: fix nuke_gone_directories, enable --delete-directories (19aae22)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 19aae22f9e7f706c2efeee0000994a0b8f1dec1d
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sat Jun 15 22:33:15 2013 -0500
umdl: fix nuke_gone_directories, enable --delete-directories
>---------------------------------------------------------------
server/update-master-directory-list | 22 +++++++++-------------
1 files changed, 9 insertions(+), 13 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 87cb6a1..24d5566 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -273,23 +273,18 @@ def make_repository(dir, dname, category, product_id, topdir):
return repo
-def nuke_gone_directories(category, category_directories):
+def nuke_gone_directories(rootdir):
""" deleting a Directory has a ripple effect through the whole
database. Be really sure you're ready do to this. It comes
in handy when say a Test release is dropped."""
- cache = setup_directory_category_cache()
- for d in list(Directory.select(orderBy='-name')):
+ for d in Directory.select(orderBy='-name'):
dname = d.name
- try:
- for c in cache[d.id]:
- if category == c and dname:
- if not category_directories.has_key(dname):
- sys.stdout.write("Deleting gone directory %s\n" % (dname))
- sys.stdout.flush()
- d.destroySelf()
- except:
- continue
+ if not os.path.isdir(os.path.join(rootdir, dname)):
+ if len(d.categories) == 1: # safety, this should always trigger
+ sys.stdout.write("Deleting gone directory %s\n" % (dname))
+ sys.stdout.flush()
+ d.destroySelf()
unreadable_dirs = set()
@@ -419,7 +414,6 @@ def sync_category_directories(category, category_directories, directory_info):
make_repository(D, dirpath, category, product_id, topdir)
make_repomd_file_details(dirpath, D)
Directory.ageFileDetails()
- nuke_gone_directories(category, category_directories)
def parse_rsync_listing(category, f):
@@ -589,6 +583,8 @@ def main():
excludes = i.get('excludes', [])
sync_directories_from_directory(i['path'], category, excludes)
+ if options.delete_directories:
+ nuke_gone_directories(options.rootdir)
remove_pidfile(pidfile)
sys.stdout.write("Ending umdl %s\n" % (datetime.datetime.utcnow().isoformat()))
sys.stdout.flush()
11 years
[mirrormanager] v1.4: use sorted() where possible in prep for python3 (3dc150a)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit 3dc150ad07d44edeb32829caea2c48af2293d66a
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sat Jun 15 20:57:57 2013 -0500
use sorted() where possible in prep for python3
>---------------------------------------------------------------
server/mirrormanager/controllers.py | 3 +--
server/mirrormanager/model.py | 10 +++-------
server/mirrormanager/templates/publiclist.kid | 3 +--
3 files changed, 5 insertions(+), 11 deletions(-)
diff --git a/server/mirrormanager/controllers.py b/server/mirrormanager/controllers.py
index ed3f226..f7cbbb5 100644
--- a/server/mirrormanager/controllers.py
+++ b/server/mirrormanager/controllers.py
@@ -929,8 +929,7 @@ class Root(controllers.RootController):
includes.update(newer_dirs)
for n in newer_dirs:
includes.update(parents(n))
- includes = list(includes)
- includes.sort()
+ includes = sorted(includes)
# add trailing slash as rsync wants it
for i in xrange(len(includes)):
includes[i] += u'/'
diff --git a/server/mirrormanager/model.py b/server/mirrormanager/model.py
index f098b45..5ac4a92 100644
--- a/server/mirrormanager/model.py
+++ b/server/mirrormanager/model.py
@@ -503,11 +503,8 @@ def _publiclist_sql_to_list(sqlresult, valid_categories):
if len(hosts[h].categories) == 0:
del hosts[h]
- # turn the dict into a list
- l = []
- for k, v in hosts.iteritems():
- l.append(v)
- l.sort()
+ # turn the dict values into a list
+ l = sorted(hosts.values())
return l
def _publiclist_hosts(product=None, re=None):
@@ -677,8 +674,7 @@ def publiclist_arches():
rc = 1
return rc
- result = [a for a in Arch.selectBy(publiclist=True)]
- result.sort(compare)
+ result = sorted(Arch.selectBy(publiclist=True), cmp=compare)
return result
# e.g. 'fedora' and 'epel'
diff --git a/server/mirrormanager/templates/publiclist.kid b/server/mirrormanager/templates/publiclist.kid
index 35028ef..016c771 100644
--- a/server/mirrormanager/templates/publiclist.kid
+++ b/server/mirrormanager/templates/publiclist.kid
@@ -75,8 +75,7 @@ and development networks globally.
<td>
<table>
<?python
- categories = host.categories.keys()
- categories.sort()
+ categories = sorted(host.categories.keys())
?>
<tr py:for="hc in categories">
<td><span py:replace="host.categories[hc].name">Category name</span></td>
11 years
[mirrormanager] v1.4: umdl: s/print/sys.stdout.write/ (d20e2e7)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit d20e2e7bee1a78a78717eba841e8f9a463bd2230
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sat Jun 15 20:38:59 2013 -0500
umdl: s/print/sys.stdout.write/
>---------------------------------------------------------------
server/update-master-directory-list | 27 ++++++++++++++++++---------
1 files changed, 18 insertions(+), 9 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 65dd641..87cb6a1 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -285,7 +285,8 @@ def nuke_gone_directories(category, category_directories):
for c in cache[d.id]:
if category == c and dname:
if not category_directories.has_key(dname):
- print "Deleting gone directory %s" % (dname)
+ sys.stdout.write("Deleting gone directory %s\n" % (dname))
+ sys.stdout.flush()
d.destroySelf()
except:
continue
@@ -442,7 +443,8 @@ def parse_rsync_listing(category, f):
def sync_directories_using_rsync(rsyncpath, category, extra_rsync_args=None):
result, output = run_rsync(rsyncpath, extra_rsync_args)
if result is False:
- print "Category %s: %s" % (category.name, output)
+ sys.stdout.write("rsync failed for Category %s: %s\n" % (category.name, output))
+ sys.stdout.flush()
return
parse_rsync_listing(category, output)
@@ -485,7 +487,7 @@ def sync_directories_from_directory(directory, category, excludes=[]):
next=True
break
if next:
- print "excluding %s" % (dirpath)
+ sys.stdout.write("excluding %s\n" % (dirpath))
# exclude all its subdirs too
dirnames[:] = []
continue
@@ -514,7 +516,8 @@ def sync_directories_from_directory(directory, category, excludes=[]):
changed = (d_ctime != ctime)
if changed:
- print "%s has changed" % dname
+ sys.stdout.write("%s has changed\n" % dname)
+ sys.stdout.flush()
category_directories[dname] = {'files':{}, 'isRepository':isRepo, 'readable':readable, 'ctime':ctime, 'changed':changed}
# skip per-file stat()s if the directory hasn't changed
@@ -549,10 +552,13 @@ def main():
delete_directories = options.delete_directories
if manage_pidfile(pidfile):
- print "another instance is running, try again later."
+ sys.stdout.write("another instance is running, try again later.\n")
+ sys.stdout.flush()
sys.exit(1)
- print "Starting umdl %s" % (datetime.datetime.utcnow().isoformat())
+
+ sys.stdout.write("Starting umdl %s\n" % (datetime.datetime.utcnow().isoformat()))
+ sys.stdout.flush()
setup_arch_version_cache()
for i in config.get('umdl.master_directories'):
try:
@@ -564,11 +570,13 @@ def main():
try:
category = Category.byName(cname)
except SQLObjectNotFound:
- print 'Error: umdl.master_directories Category %s does not exist in the database, skipping' % (cname)
+ sys.stdout.write('Error: umdl.master_directories Category %s does not exist in the database, skipping\n' % (cname))
+ sys.stdout.flush()
continue
if category.product is None:
- print 'Error: umdl.master_directories Category %s has null Product, skipping' % (cname)
+ sys.stdout.write('Error: umdl.master_directories Category %s has null Product, skipping\n' % (cname))
+ sys.stdout.flush()
continue
if i['type'] == 'rsync':
@@ -582,7 +590,8 @@ def main():
sync_directories_from_directory(i['path'], category, excludes)
remove_pidfile(pidfile)
- print "Ending umdl %s" % (datetime.datetime.utcnow().isoformat())
+ sys.stdout.write("Ending umdl %s\n" % (datetime.datetime.utcnow().isoformat()))
+ sys.stdout.flush()
return 0
11 years
[mirrormanager] v1.4: umdl: don't make duplicate Repository objects (d03bebd)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit d03bebd177cd12d3f7091ec52d075942ad4c5191
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sat Jun 15 20:31:37 2013 -0500
umdl: don't make duplicate Repository objects
While it's not completely illegal to have 2 or more Repository objects
pointing at the same Directory (witness the old Fedora Core pointers,
which had two different mirrorlist request URLs, one stating fc<#> and
one stating f<#> - in general we don't need them, and in specific we
don't want umdl automatically making them. So don't let it.
>---------------------------------------------------------------
server/update-master-directory-list | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 1e8b20d..65dd641 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -250,6 +250,10 @@ def make_repository(dir, dname, category, product_id, topdir):
if ver is None or arch is None:
return None
+ # stop making duplicate Repository objects.
+ if len(dir.repositories) > 0:
+ return None
+
prefix = repo_prefix(path, category, ver)
try:
# historically, Repository.name was a longer string with
11 years
[mirrormanager] v1.4: umdl: return the right type of object (b2f6924)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : v1.4
>---------------------------------------------------------------
commit b2f6924fd9a493aa190ddfabcc054ca806fd320b
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sat Jun 15 02:44:25 2013 -0500
umdl: return the right type of object
>---------------------------------------------------------------
server/update-master-directory-list | 14 +++++++-------
1 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index 8c37d42..1e8b20d 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -115,13 +115,13 @@ def guess_ver_arch_from_path(category, product_id, path):
# create Versions if we can figure it out...
if ver is None:
- ver = create_version_from_path(category, path)
- if ver:
- version_cache.append(dict(id = ver.id,
- name = ver.name,
- product_id = product_id,
- ordered_mirrorlist = ver.ordered_mirrorlist)
- )
+ v = create_version_from_path(category, path)
+ if v:
+ ver = dict(id = v.id, name = v.name,
+ product_id = product_id,
+ ordered_mirrorlist = v.ordered_mirrorlist)
+
+ version_cache.append(ver)
return (ver, arch)
11 years