On Thu, Feb 22, 2007 at 02:23:57PM -0700, Jonathan Steffan wrote:
Team,
I've been working on getting jigdo usable for Fedora Unity. In the
process, I needed a way to have a single url to access rpms on public
mirrors.
Mirrormanager has code to do this right now if you want it. (Granted,
it's not in use yet, but soon...). The URLs underneath /pub/.... on
the mirrormanager URL provide the list of up-to-date mirrors that have
that content. Right now it's all directory-based, but could easily be
extended to be file-based. By that I mean:
http://admin.fedora.redhat.com/mirrormanager/pub/fedora/linux/core/6/i386...
will return the list of mirrors containing the content of
pub/fedora/linux/core/6/i386/iso/
that's up-to-date.
Assume the requested file is foo.rpm What I have done so far is
setup a rewrite map that does the following:
* Pull the mirrorlist while preserving the requesting IP as
X-Fowarded-For (for geoIP)
* Parse the mirrorlist
* Random loop the mirrorlist and request the HEAD for foo.rpm from a
given mirror, if not 404 continue
Is the code for this piece available? Does it use keepalives? :-)
The keepalives are what tried to hack into the mirrormanager crawler
earlie this week. (urlgrabber's keepalive.py is close to what I need,
but I need it to do HEADs not GETs, so I was overriding various parts
and it got messy in a hurry).
* Redirect the request for foo.rpm to a public mirror (using
302) that
has verified it has the file
Things I plan on doing:
* Cache results (assuming no DB backend, do RAM caching based on
session; IP, remote mirror, etc)
* Maybe rate public mirrors based on:
* Number of missing files (404)
* Latency (granted this is from the rewrite server)
* Bitrate test (ran every hour or something, also from the rewrite
server)
* Setup a database that can be prepopulated with this data, potentially
using data from mirrormanager
:-)
If needed, round robin would help keep things going.. or even just use
pound (or something else) between two machines.
Any thoughts? I'd like to have Fedora Project to provide this feature. If
not, I will be setting it up anyways.
Jonathan Steffan
daMaestro
if mirrormanager can provide what you need, I'm sure open to
contributions (ideas and/or patches).
Thanks,
Matt
--
Matt Domsch
Software Architect
Dell Linux Solutions
linux.dell.com &
www.dell.com/linux
Linux on Dell mailing lists @
http://lists.us.dell.com