On Fri, Jun 26, 2015 at 06:11:44PM +0200, Pierre-Yves Chibon wrote:
Yesterday and today I spent a little time going over the UDML script
Going through it, I ended up with few questions regarding it.
* Repository name
UMDL's code clearly says:
# historically, Repository.name was a longer string with
# product and category deliniations. But we were getting
# unique constraint conflicts once we started introducing
# repositories under repositories. And .name isn't used for
# anything meaningful. So simply have it match dir.name,
# which can't conflict.
And quickly grepping through MM2's sources, I could not find a reference to
this, we alway rely on the repository's prefix, not its name.
Question: Should we drop this?
It makes things confusing and is basically noise since we do not use it anywhere.
It was a helpful column for fixing errors with the repos. But as the
database is so huge everything we could drop should be dropped.
* The directory table
So looking at the database and more precisely the directory table in that
database, it seems we store all the directories of the tree, ie:
This makes me a little pondering. What is the interest of keeping the whole
list of directories in the DB ?
After all, as far as I understand, the UMDL finds the repo in the tree (repo
being defined by the presence of a 'repodata' folder containing the repomd.xml
or by the presence of a 'summary' file and an 'objects' folder).
For these repo, we look for the most recent files, stores this info in the DB
and later use it to check if the mirrors are up to date.
But do we need to checking that ``pub/fedora/linux`` exists when we later check
that ``pub/fedora/linux/updates/testing/21/x86_64/`` exists and is up to date?
I am under the impression currently that dropping un-necessary directories would
save DB space (the directories being then linked in the host_category_dir table
listing for each host, in each category which dir are present) as well as
crawling time (both in the UMDL and in the crawler).
Again, dropping unnecessary information from the database sounds good.
Although this one sounds a bit more complex as you always have to delete
directories if subdirectories appear and add directories if
* Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree:
We, in Fedora, are only using the last one. I believe the `rsync` mode was added
to support Ubuntu and the file mode is basically a simplified version of the
directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be
simpler and easier to create an UMDL and a crawler for each distro that would
like to use MirrorManager than maintaining a one-script-fits-all UMDL that is
in fact tested for only one of the scenario.
That being said, if we ever have interest from Ubuntu, CentOS or any other
communities, we should definitively look into making the UMDL and crawler as
re-usable as possible for them, but keeping the distro-specific bits separated.
Like already mentioned, RPM Fusion uses the rsync mode as the master
mirror is 'far' away from the MirrorManager installation. It is still
using MM1 on CentOS 5 and currently I am not immediately planing on
upgrading to MM2. So it could be removed and I should be able to write
the necessary umdl rsync crawler once I need it.
Another thought about umdl I had concerns the file mode. We have for
the categories 'Fedora EPEL' and 'Fedora Linux' files called
'fullfilelist'. Maybe that would be an option for umdl to use to reduce
I/O on the NFS mounts. Only actually reading the files and metadata from
NFS if it is necessary. Just one of those ideas.