On Sat, May 03, 2014 at 11:40:49PM +0200, Pierre-Yves Chibon wrote:
I would like to run by you my understanding of how mirrormanager works, I hope
I am not to far from the truth but please if I am let me know :)
Mirrormanager is splitted into three parts: the UI, the API and a cron task.
There are several cronjobs, but basically correct.
The UI is the current TurboGears1 application . People can login
in there and
register an institution (Site) with one or more sub-domain (Host) each mirroring
one or more Product (Fedora, EPEL...).
For each sub-domain (Host), there are a number of settings available such as
where the mirror pulls its updates, who can pull from it, restricting the user's
country or network.
Users manage their mirror there, admin can access any Site/Host and update their
settings as well.
Correct. This is the ugliest part of MM1.x, I will be most glad for
its complete rewrite.
The API is a cgi script, called in our case by yum, which redirects
the user to
the closest mirror, active, up-to-date and according to the settings of the
Correct. This (mirrorlist-server directory) is almost entirely
divorsed from the TG 1.x components. It does not touch the database
directly in any way, but is handed a pickle containing a precalculated
cache of the database, which can then be distributed to each of the
mirrorlist-server servers. As long as you don't break the format of
the pickle, you don't have to change this code.
The cron task, runs daily and crawls the mirror to check if they are
up to date or not and register in the database which folder are up
to date and which are not for every single mirror. This cron task
(or is it another?) also generates the publiclist pages  using
the data it just retrieved about each mirror.
There are several different cronjobs.
1) crawlers which checks every directory on every mirror every few
(~2-4) hours, and updates the database with which directories on each
mirror are up-to-date. We don't mark whole servers as stale, but
individual directories on individual servers.
2) generate the publiclist pages.
3) generate the mirrorlist-server pickle.
4) retrieve some external network routing data from public routers
(both Internet2 and Internet1) used to identify which servers and
clients are on which autonomous systems, and to select the proper
5) get the monthly GeoIP database from maxmind. This isn't part of MM
proper but is necessary to run.
I have been trying to figure out how exactly are retrieved the
publiclist. I found the query used by MM1  and I was wondering if there would
not be a way to simplify that. Would it be an option to have the cron task
setting a flag on the database saying if a host is up to date or not?
Or am I missing some of the information?
It is complicated because there are two main cases:
1) we track mirror freshness by individual directory, not by whole
mirror. We will put a mirror on the publiclist if at least one of its
directories has up-to-date content. The liklihood of a mirror having
at least one directory stale throughout any given day is quite high as
content changes on the masters frequently; I don't want to keep
adding and removing mirrors from the publiclist at that frequency.
To change to a host-global up-to-date flag would certainly simplify
this logic, at a loss of granularity.
2) a few mirrors are special, they have the 'always_up2date' flag set
on their HostCategory. These are particularly Fedora master mirrors
that we don't want to crawl, but do want them to appear in the
publiclist. A second case is "trusted" mirrors (e.g. the ones I run
at Dell) which because of firewalls cannot be hit by the crawlers, but
can serve content internally just fine. Those are also marked
"always_up2date" and it's incumbant on me to make that assertion true.
If we could simplify this part we could drop the cron task generating
publiclist pages and just display them on the fly as part of the UI.
I'd prefer not, but I'll leave that to the collective new maintainance
team to determine the desired output. :-)
As part of the re-write, there is of course the UI since it is
the CGI script and the cron tasks should not need much changes, would they?
Some of the cron tasks call into the TG1 model/controller code directly, so yes they
would. The CGI would not.
There are a few other "helper apps" like
which also call into the TG1 model/controller code and would need to