[mirrormanager] master: remove TODO items that are complete (ae6726d)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit ae6726df3d4a3f8e83dab225ef1994bddc2c4729
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sun Dec 22 15:16:17 2013 -0600
remove TODO items that are complete
>---------------------------------------------------------------
TODO | 11 -----------
1 files changed, 0 insertions(+), 11 deletions(-)
diff --git a/TODO b/TODO
index 201e14c..38d5e19 100644
--- a/TODO
+++ b/TODO
@@ -5,22 +5,11 @@ Short Term:
* add metalink/HTTP support, handy for what otherwise is a redirect to a single mirror
-* Better theming support. Right now it's very Fedora-centric in the
- UI, and there are various "if project==rpmfusion" kinds of things
- scattered throughout. Ugly. Make it easy to point to custom css and
- custom theme, so the Fedora theme becomes just separate theme.
-
-* clean up add-instrepos and mv-instrepos - ideally umdl would handle these.
* move Fedora-specific apps into a server/fedora directory, add /usr/share/mirrormanager/server to sys.path.
* crawlers failing after a 90min timeout - why?
-* consider making http redirects only redirect to http servers, not FTP (fixes bfo/ipxe)
Long Term:
-* Let report_mirror work from behind an http proxy. It presently
- doesn't try to connect to the proxy server.
- - httplib + libcurl please
-
* send emails to mirror admins when their servers drop out of the
mirrorlists, with why.
10 years, 4 months
[mirrormanager] master: Merge branch 'master' of ssh+git://fedorahosted.org/git/mirrormanager (0a74325)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit 0a743253307af1781e307340f22e2e9da8637a00
Merge: 84b61ec b4d6a13
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sun Dec 22 14:57:36 2013 -0600
Merge branch 'master' of ssh+git://fedorahosted.org/git/mirrormanager
>---------------------------------------------------------------
server/mirrormanager/mirrorlist.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/mirrormanager/mirrorlist.py b/server/mirrormanager/mirrorlist.py
index e67c9c2..3618c39 100644
--- a/server/mirrormanager/mirrorlist.py
+++ b/server/mirrormanager/mirrorlist.py
@@ -124,7 +124,7 @@ def populate_directory_cache():
continue
if directoryname not in cache:
- cache[directoryname] = {'global':set(), 'byCountry':{}, 'byHostId':{}, 'ordered_mirrorlist':False, 'byCountryInternet2':{}}
+ cache[directoryname] = {'global':set(), 'byCountry':{}, 'byHostId':{}, 'ordered_mirrorlist':True, 'byCountryInternet2':{}}
repo = directory_repo_cache.get(directory_id)
10 years, 4 months
[mirrormanager] master: mirrorlist_client: fix mirrorlist/ early return (84b61ec)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit 84b61ecd465349860ad177722f819bc1b473b2b7
Author: Matt Domsch <Matt_Domsch(a)dell.com>
Date: Sun Dec 22 14:56:14 2013 -0600
mirrorlist_client: fix mirrorlist/ early return
ec0c2db90452367eae2567b1cc1401ad9f2b1a78
introduced a failure for all mirrorlist/ requests, prematurely
returning. The root cause was an indentation problem.
>---------------------------------------------------------------
mirrorlist-server/mirrorlist_client.wsgi | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index a4853e9..7250ea4 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -170,7 +170,7 @@ def application(environ, start_response):
response.headers['Location'] = str(url)
else:
response.status_code=404
- return response(environ, start_response)
+ return response(environ, start_response)
text = ""
text += message + '\n'
10 years, 4 months
[mirrormanager] master: default directories to ordered_mirrorlist = True (b4d6a13)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit b4d6a13208c37db15158876606530bd33eb0d1b7
Author: Matt Domsch <matt(a)domsch.com>
Date: Sun Dec 22 14:10:13 2013 -0600
default directories to ordered_mirrorlist = True
Reported by two private mirror operators on mirror-list-d and
privately, mirror operators using metalinks were getting _only_ their
mirror returned in the metalink, not theirs, plus several fallbacks.
When their mirrors then were not working or not up-to-date or not
reachable, yum would fail because there were no more mirrors to try.
Complicating things, when viewed using mirrorlist/ rather than
metalink/, mirrorlist returned a long list, while metalink didn't.
The proximate cause is that mirrorlist/ looks at Repository pointers
into a directory, where the ordered_mirrorlist=True value is set on
the corresponding Version object for that Repository. However,
metalink/ was looking one directory deeper, at the repodata/
directory, which was _never_ being treated as
ordered_mirrorlist=True. This causes the path through
mirrorlist_server.py:do_mirrorlist() to take the !ordered_mirrorlist
path, which was intended only for long-obsolete product Versions
We should delete the whole !ordered_mirrorlist codepath, but won't do
so at this time.
Instead, we'll solve the immediate problem with the smallest possible
change, default ordered_mirrorlist=True for all directories, and only
historical (non-metalink-using) Versions may set it to False.
>---------------------------------------------------------------
server/mirrormanager/mirrorlist.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/mirrormanager/mirrorlist.py b/server/mirrormanager/mirrorlist.py
index e67c9c2..3618c39 100644
--- a/server/mirrormanager/mirrorlist.py
+++ b/server/mirrormanager/mirrorlist.py
@@ -124,7 +124,7 @@ def populate_directory_cache():
continue
if directoryname not in cache:
- cache[directoryname] = {'global':set(), 'byCountry':{}, 'byHostId':{}, 'ordered_mirrorlist':False, 'byCountryInternet2':{}}
+ cache[directoryname] = {'global':set(), 'byCountry':{}, 'byHostId':{}, 'ordered_mirrorlist':True, 'byCountryInternet2':{}}
repo = directory_repo_cache.get(directory_id)
10 years, 4 months
[mirrormanager] master: mirrorlist_client: don't return redirects to non-HTTP servers (ec0c2db)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit ec0c2db90452367eae2567b1cc1401ad9f2b1a78
Author: Matt Domsch <matt(a)domsch.com>
Date: Wed Dec 18 22:57:33 2013 -0600
mirrorlist_client: don't return redirects to non-HTTP servers
https://fedorahosted.org/fedora-infrastructure/ticket/3610
>---------------------------------------------------------------
mirrorlist-server/mirrorlist_client.wsgi | 20 +++++++++++++++-----
1 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/mirrorlist-server/mirrorlist_client.wsgi b/mirrorlist-server/mirrorlist_client.wsgi
index 5f49cda..a4853e9 100755
--- a/mirrorlist-server/mirrorlist_client.wsgi
+++ b/mirrorlist-server/mirrorlist_client.wsgi
@@ -134,6 +134,13 @@ def manage_compression(request, response, buf):
response.headers['Content-Encoding'] = 'gzip'
return buf
+def keep_only_http_results(input):
+ output = []
+ for hostid, url in input:
+ if url.startswith(u'http'):
+ output.append((hostid, url))
+ return output
+
def application(environ, start_response):
request = WSGIRequest(environ)
response = WSGIResponse()
@@ -155,12 +162,15 @@ def application(environ, start_response):
if 'redirect' in request.GET:
if len(results) == 0:
response.status_code=404
- return response(environ, start_response)
else:
- (hostid, url) = results[0]
- response.status_code=302
- response.headers['Location'] = str(url)
- return response(environ, start_response)
+ results = keep_only_http_results(results)
+ if len(results):
+ (hostid, url) = results[0]
+ response.status_code=302
+ response.headers['Location'] = str(url)
+ else:
+ response.status_code=404
+ return response(environ, start_response)
text = ""
text += message + '\n'
10 years, 4 months
[mirrormanager] master: crawler_perhost: fix hosts crawled via rsync (7216df0)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit 7216df04935bb52d104cbf26f60dfbd4bacf94a0
Author: Matt Domsch <matt(a)domsch.com>
Date: Wed Dec 18 22:39:00 2013 -0600
crawler_perhost: fix hosts crawled via rsync
Hosts crawled via rsync were being reported as having content
they do not have. Turns out, I neglected to set the all_files
flag to False when the files were missing. This fixes that mistake.
>---------------------------------------------------------------
server/crawler_perhost | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/server/crawler_perhost b/server/crawler_perhost
index 199e3e1..9510feb 100755
--- a/server/crawler_perhost
+++ b/server/crawler_perhost
@@ -549,6 +549,7 @@ def try_percategory(trydirs, url, host_category_dirs, hc, host, categoryPrefixLe
except KeyError: # file is not in the rsync listing
msg = 'Missing remote file %s\n' % key
logger.debug(msg)
+ all_files = False
break
except: # something else went wrong
exception_msg = "Exception caught in try_percategory()\n"
10 years, 4 months
[mirrormanager] master: silence cronjobs - they are already writing logs (afb9fee)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit afb9feeae9808f5420fe1ee7702316e4546e1046
Author: Matt Domsch <matt(a)domsch.com>
Date: Wed Dec 18 18:39:24 2013 -0600
silence cronjobs - they are already writing logs
>---------------------------------------------------------------
cron/mirrormanager.cron | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/cron/mirrormanager.cron b/cron/mirrormanager.cron
index e089fef..d5f92fa 100644
--- a/cron/mirrormanager.cron
+++ b/cron/mirrormanager.cron
@@ -2,14 +2,14 @@
# adjust frequency according to how often your content changes
#
# refresh the mirrorlist cache at the top of the hour
-55 * * * * mirrormanager /usr/share/mirrormanager/server/update-mirrorlist-server >> /var/log/mirrormanager/mirrorlist.log 2>&1
+55 * * * * mirrormanager /usr/share/mirrormanager/server/update-mirrorlist-server > /dev/null 2>&1
# refresh the publiclist static pages daily
-55 0 * * * mirrormanager /usr/share/mirrormanager/server/update-static-content >> /var/log/mirrormanager/static-content.log 2>&1
+55 0 * * * mirrormanager /usr/share/mirrormanager/server/update-static-content > /dev/null 2>&1
# update master directory list
# logs sent to /var/log/mirrormanager/umdl.log by default
-0 */2 * * * mirrormanager cd /usr/share/mirrormanager/server && ./update-master-directory-list -c /etc/mirrormanager/prod.cfg > /dev/null
+0 */2 * * * mirrormanager cd /usr/share/mirrormanager/server && ./update-master-directory-list -c /etc/mirrormanager/prod.cfg > /dev/null 2>&1
# Get the global netblocks list once a day
30 0 * * * mirrormanager cd /usr/share/mirrormanager/server && ./get_global_netblocks /var/lib/mirrormanager/global_netblocks.txt > /dev/null 2>&1
10 years, 4 months
[mirrormanager] master: model.py: fix report_mirror checkin (e40e13a)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit e40e13aee30d03b0a18d7b31eff609b3ee6cc127
Author: Matt Domsch <matt(a)domsch.com>
Date: Wed Dec 18 16:08:38 2013 -0600
model.py: fix report_mirror checkin
>---------------------------------------------------------------
server/mirrormanager/model.py | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/server/mirrormanager/model.py b/server/mirrormanager/model.py
index 075cc81..8ea815e 100644
--- a/server/mirrormanager/model.py
+++ b/server/mirrormanager/model.py
@@ -323,7 +323,7 @@ class Host(SQLObject):
def _set_config(self, config):
# really, we don't store the config anymore
self._config = None
- self.lastCheckedIn = datetime.datetime.utcnow
+ self.lastCheckedIn = datetime.datetime.utcnow()
def checkin(self, config):
message = self._uploaded_config(config)
10 years, 4 months
[mirrormanager] master: Directory.destroySelf(): handle HostCategoryDir deletion (b5cef98)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit b5cef9807b7becaaca8158680b33481ffd7261c0
Author: Matt Domsch <matt(a)domsch.com>
Date: Wed Dec 18 15:33:41 2013 -0600
Directory.destroySelf(): handle HostCategoryDir deletion
The crawler may create or destroy these items independently,
and concurrent to umdl invoking Directory.destroySelf().
Don't fail to destroy these just because something else already did.
>---------------------------------------------------------------
server/mirrormanager/model.py | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/server/mirrormanager/model.py b/server/mirrormanager/model.py
index 961cf10..075cc81 100644
--- a/server/mirrormanager/model.py
+++ b/server/mirrormanager/model.py
@@ -686,7 +686,10 @@ class Directory(SQLObject):
r.destroySelf()
# don't destroy a whole category if only deleting a directory
for hcd in self.host_category_dirs:
- hcd.destroySelf()
+ # try/except this because crawlers running in parallel may
+ # create or destroy these at will
+ try: hcd.destroySelf()
+ except: pass
for fd in self.fileDetails:
fd.destroySelf()
for eh in self.exclusive_hosts:
10 years, 4 months
[mirrormanager] master: umdl: category.directories is a list, treat it as such (259c974)
by Matt Domsch
Repository : http://git.fedorahosted.org/cgit/
On branch : master
>---------------------------------------------------------------
commit 259c9747b3f74686a31f82c77059bb78423e78f2
Author: Matt Domsch <matt(a)domsch.com>
Date: Wed Dec 18 15:30:53 2013 -0600
umdl: category.directories is a list, treat it as such
>---------------------------------------------------------------
server/update-master-directory-list | 4 +++-
1 files changed, 3 insertions(+), 1 deletions(-)
diff --git a/server/update-master-directory-list b/server/update-master-directory-list
index f92ee6d..25583fb 100755
--- a/server/update-master-directory-list
+++ b/server/update-master-directory-list
@@ -274,7 +274,9 @@ def nuke_gone_directories(diskpath, category):
topdirName = category.topdir.name
- for d in category.directories(orderBy='-name'):
+ directories = category.directories # in ascending name order
+ directories.reverse() # now in decending name order, bottoms up
+ for d in directories:
if is_excluded(d.name, category.excludes): continue
relativeDName = remove_category_topdir(topdirName, d.name)
if not os.path.isdir(os.path.join(diskpath, relativeDName)):
10 years, 4 months