Dear all,
Yesterday and today I spent a little time going over the UDML script of MirrorManager2. Going through it, I ended up with few questions regarding it.
* Repository name UMDL's code clearly says: # historically, Repository.name was a longer string with # product and category deliniations. But we were getting # unique constraint conflicts once we started introducing # repositories under repositories. And .name isn't used for # anything meaningful. So simply have it match dir.name, # which can't conflict. And quickly grepping through MM2's sources, I could not find a reference to this, we alway rely on the repository's prefix, not its name.
Question: Should we drop this? It makes things confusing and is basically noise since we do not use it anywhere.
* Readable status of directories The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
* Changes while running Looking at the code, the UMDL seems to be very careful to handle changes on the FS while it is running. One hope I have is to speed up the UMDL run time, but I'm curious.
Question: Does anyone know if the FS changes often while the UMDL is actually running? Gaining speed of course does not mean being wreakless but I'm curious as to how often this situation occurs. IIRC, we trigger the UMDL via fedmsg now, right? So in theory, the FS shouldn't change too much under the UMDL's feet.
* The directory table So looking at the database and more precisely the directory table in that database, it seems we store all the directories of the tree, ie: /pub/alt/ /pub/alt/anaconda/ /pub/alt/bfo/ /pub/alt/bfo/gpxe-20120514 ... This makes me a little pondering. What is the interest of keeping the whole list of directories in the DB ? After all, as far as I understand, the UMDL finds the repo in the tree (repo being defined by the presence of a 'repodata' folder containing the repomd.xml or by the presence of a 'summary' file and an 'objects' folder). For these repo, we look for the most recent files, stores this info in the DB and later use it to check if the mirrors are up to date.
But do we need to checking that ``pub/fedora/linux`` exists when we later check that ``pub/fedora/linux/updates/testing/21/x86_64/`` exists and is up to date?
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in the crawler).
* Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree: * file * rsync * directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
Looking forward hearing your thoughts about these points and questions, Thanks,
Pierre
On Fri, 26 Jun 2015 18:11:44 +0200 Pierre-Yves Chibon pingou@pingoured.fr wrote:
Dear all,
Yesterday and today I spent a little time going over the UDML script of MirrorManager2. Going through it, I ended up with few questions regarding it.
- Repository name
UMDL's code clearly says: # historically, Repository.name was a longer string with # product and category deliniations. But we were getting # unique constraint conflicts once we started introducing # repositories under repositories. And .name isn't used for # anything meaningful. So simply have it match dir.name, # which can't conflict. And quickly grepping through MM2's sources, I could not find a reference to this, we alway rely on the repository's prefix, not its name.
Question: Should we drop this? It makes things confusing and is basically noise since we do not use it anywhere.
Sounds fine to me if we don't use it.
- Readable status of directories
The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
Could it be for when we have a release about to come out, but it's not readable to the public yet? Typically we stage a release on friday and until the actual release on tuesday the directory isn't open, only mirrors with the acls can sync it.
- Changes while running
Looking at the code, the UMDL seems to be very careful to handle changes on the FS while it is running. One hope I have is to speed up the UMDL run time, but I'm curious.
Question: Does anyone know if the FS changes often while the UMDL is actually running? Gaining speed of course does not mean being wreakless but I'm curious as to how often this situation occurs. IIRC, we trigger the UMDL via fedmsg now, right? So in theory, the FS shouldn't change too much under the UMDL's feet.
Well, I can think of one common case:
1. Fedora updates push finishes, umdl starts. 2. EPEL updates push finishes while umdl is in the middle of it's directories.
We could of course fix this by making it crawl them seperately? For each category?
- The directory table
So looking at the database and more precisely the directory table in that database, it seems we store all the directories of the tree, ie: /pub/alt/ /pub/alt/anaconda/ /pub/alt/bfo/ /pub/alt/bfo/gpxe-20120514 ... This makes me a little pondering. What is the interest of keeping the whole list of directories in the DB ? After all, as far as I understand, the UMDL finds the repo in the tree (repo being defined by the presence of a 'repodata' folder containing the repomd.xml or by the presence of a 'summary' file and an 'objects' folder). For these repo, we look for the most recent files, stores this info in the DB and later use it to check if the mirrors are up to date.
But do we need to checking that ``pub/fedora/linux`` exists when we later check that ``pub/fedora/linux/updates/testing/21/x86_64/`` exists and is up to date?
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in the crawler).
Yeah, I can't think of a reason we keep all that.
- Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree:
- file
- rsync
- directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
Sounds fine to me, although it seems rpmfusion might use the rsync one.
Looking forward hearing your thoughts about these points and questions, Thanks,
I had one additional thought based out of recent issues we have had:
Right now when an updates push happens, umdl starts and crawls everything in all directory trees. Perhaps we could be much more targted here? if the fedmsg says 'rawhide' was updated, only crawl that area. If it says "Fedora 21" or "EPEL 7" only crawl those.
And then of course we would need some way to have it crawl everything in order to add new releases like Fedora 23 Alpha or whatever, but it could just do that once a day? Or on demand?
Just a thought.
kevin
On Fri, Jun 26, 2015 at 10:50:07AM -0600, Kevin Fenzi wrote:
Yesterday and today I spent a little time going over the UDML script of MirrorManager2. Going through it, I ended up with few questions regarding it.
[...]
- Readable status of directories
The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
Could it be for when we have a release about to come out, but it's not readable to the public yet? Typically we stage a release on friday and until the actual release on tuesday the directory isn't open, only mirrors with the acls can sync it.
Yes, that's also what I remember about it. Although I am not sure if we still need it. As we bitflip much earlier we have the chance to crawl everything before the release. The last few releases we even didn't use the data under releases but under development for the first few weeks. So this functionality has become pretty useless with the current release mechanism.
- Changes while running
Looking at the code, the UMDL seems to be very careful to handle changes on the FS while it is running. One hope I have is to speed up the UMDL run time, but I'm curious.
Question: Does anyone know if the FS changes often while the UMDL is actually running? Gaining speed of course does not mean being wreakless but I'm curious as to how often this situation occurs. IIRC, we trigger the UMDL via fedmsg now, right? So in theory, the FS shouldn't change too much under the UMDL's feet.
Well, I can think of one common case:
- Fedora updates push finishes, umdl starts.
- EPEL updates push finishes while umdl is in the middle of it's
directories.
Yes, the data definitely changes during umdl's runtime.
We could of course fix this by making it crawl them seperately? For each category?
+1
[...]
I had one additional thought based out of recent issues we have had:
Right now when an updates push happens, umdl starts and crawls everything in all directory trees. Perhaps we could be much more targted here? if the fedmsg says 'rawhide' was updated, only crawl that area. If it says "Fedora 21" or "EPEL 7" only crawl those.
And then of course we would need some way to have it crawl everything in order to add new releases like Fedora 23 Alpha or whatever, but it could just do that once a day? Or on demand?
Just a thought.
I was having the same idea. Not as fine grained as yours but similar. UMDL crawling EPEL is really fast and Fedora Others takes many hours. For 'Fedora Linux' and 'Fedora EPEL' we have working fedmsg triggers so we should definitely crawl each category separately and only on demand.
These is still this ticket https://fedorahosted.org/rel-eng/ticket/6157 If we could get a fedmsg trigger for Fedora Secondary we only would need to crawl Fedora Other maybe once per day and Fedora Archive could even be changed to manually after a release has been moved. This would reduce the I/O load on the NFS and make changes to fast changing categories like 'Fedora Linux' and 'Fedora EPEL' faster available in the database.
So this is something I will implement next week as far as it is now possible.
One the main problem I see with MirrorManager is that the number of mirrors and the data served by MirrorManager has grown enormously so that we have to get much smarter at many places to scale reasonably.
Adrian
On Fri, 26 Jun 2015 21:00:37 +0200 Adrian Reber adrian@lisas.de wrote:
...snip...
- Readable status of directories
The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
Could it be for when we have a release about to come out, but it's not readable to the public yet? Typically we stage a release on friday and until the actual release on tuesday the directory isn't open, only mirrors with the acls can sync it.
Yes, that's also what I remember about it. Although I am not sure if we still need it. As we bitflip much earlier we have the chance to crawl everything before the release. The last few releases we even didn't use the data under releases but under development for the first few weeks. So this functionality has become pretty useless with the current release mechanism.
I'm not sure I agree. ;)
We stage a release friday with the directory closed. True, we have been doing the bit-flip sooner, but usually it's late monday night so more mirrors will be synced for the release tuesday morning.
So, if we remove this, what happens friday->monday night? ie, what does property do? Does it use the info for seeing if something is up to date, or just if it's set it doesn't give out metalinks with those paths? or ?
kevin
On Fri, Jun 26, 2015 at 01:45:01PM -0600, Kevin Fenzi wrote:
On Fri, 26 Jun 2015 21:00:37 +0200 Adrian Reber adrian@lisas.de wrote:
...snip...
- Readable status of directories
The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
Could it be for when we have a release about to come out, but it's not readable to the public yet? Typically we stage a release on friday and until the actual release on tuesday the directory isn't open, only mirrors with the acls can sync it.
Yes, that's also what I remember about it. Although I am not sure if we still need it. As we bitflip much earlier we have the chance to crawl everything before the release. The last few releases we even didn't use the data under releases but under development for the first few weeks. So this functionality has become pretty useless with the current release mechanism.
I'm not sure I agree. ;)
We stage a release friday with the directory closed. True, we have been doing the bit-flip sooner, but usually it's late monday night so more mirrors will be synced for the release tuesday morning.
So, if we remove this, what happens friday->monday night? ie, what does property do? Does it use the info for seeing if something is up to date, or just if it's set it doesn't give out metalinks with those paths? or ?
Matt, do you remember how the 'readable' property works?
Thanks, Pierre
Dell Customer Communication
-- Matt Domsch Senior Distinguished Engineer & Executive Director Dell | Software Group, Office of the CTO
-----Original Message----- From: Pierre-Yves Chibon [mailto:pingou@pingoured.fr] Sent: Monday, June 29, 2015 4:19 AM To: Fedora Infrastructure Cc: Domsch, Matt Subject: Re: Thoughts and question about MM2's UMDL script
On Fri, Jun 26, 2015 at 01:45:01PM -0600, Kevin Fenzi wrote:
On Fri, 26 Jun 2015 21:00:37 +0200 Adrian Reber wrote:
...snip...
- Readable status of directories The Directory table has a
'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
Could it be for when we have a release about to come out, but it's not readable to the public yet? Typically we stage a release on friday and until the actual release on tuesday the directory isn't open, only mirrors with the acls can sync it.
Yes, that's also what I remember about it. Although I am not sure if we still need it. As we bitflip much earlier we have the chance to crawl everything before the release. The last few releases we even didn't use the data under releases but under development for the first few weeks. So this functionality has become pretty useless with the current release mechanism.
I'm not sure I agree. ;)
We stage a release friday with the directory closed. True, we have been doing the bit-flip sooner, but usually it's late monday night so more mirrors will be synced for the release tuesday morning.
So, if we remove this, what happens friday->monday night? ie, what does property do? Does it use the info for seeing if something is up to date, or just if it's set it doesn't give out metalinks with those paths? or ?
Matt, do you remember how the 'readable' property works?
== MD == Recall there are 2 ways a host_category_directory entry gets created. 1) crawler; 2) report_mirror. Readable=False is a flag so that crawler doesn't mark directories as not up2date or mark for deletion just because it can't read them pre-bitflip, having been created by report_mirror. In a sense, HCDs pointing to directories with readable=False is foretelling the future state of the mirror list (what will be reported as soon as the bitflip happens and UMDL sets readable=True) without having to wait for the crawler to hit a given mirror or for that mirror to run report_mirror again.
On 26 June 2015 at 10:11, Pierre-Yves Chibon pingou@pingoured.fr wrote:
Dear all,
So the UMDL script currently supports three ways of crawling the tree:
- file
- rsync
- directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
I thought rsync was added for speed versus Ubuntu and that we actually use it a lot on the crawlers... because that is the reason we needed to move the memory usage up.. it uses a lot of memory but takes a lot shorter period to 'crawl' or am I conflating two different things?
Also the CentOS guys have expressed interest in using mirrormanager in the past (like 6-9 months ago?) but I don't know their present interest.
Looking forward hearing your thoughts about these points and questions, Thanks,
Pierre
infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
On Fri, Jun 26, 2015 at 11:04:35AM -0600, Stephen John Smoogen wrote:
On 26 June 2015 at 10:11, Pierre-Yves Chibon pingou@pingoured.fr wrote:
So the UMDL script currently supports three ways of crawling the tree:
- file
- rsync
- directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
I thought rsync was added for speed versus Ubuntu and that we actually use it a lot on the crawlers... because that is the reason we needed to move the memory usage up.. it uses a lot of memory but takes a lot shorter period to 'crawl' or am I conflating two different things?
That's the rsync of the crawler scanning the mirrors, that's much faster compared to http/ftp. umdl analyzing the master mirror can either use the local filesystem or rsync or a file as input.
Adrian
Comments in == MD ==.
-- Matt Domsch Senior Distinguished Engineer & Executive Director Dell | Software Group, Office of the CTO
________________________________________ From: Pierre-Yves Chibon [pingou@pingoured.fr] Sent: Friday, June 26, 2015 11:11 AM To: Fedora Infrastructure Cc: Domsch, Matt Subject: Thoughts and question about MM2's UMDL script
Dear all,
Yesterday and today I spent a little time going over the UDML script of MirrorManager2. Going through it, I ended up with few questions regarding it.
* Repository name UMDL's code clearly says: # historically, Repository.name was a longer string with # product and category deliniations. But we were getting # unique constraint conflicts once we started introducing # repositories under repositories. And .name isn't used for # anything meaningful. So simply have it match dir.name, # which can't conflict. And quickly grepping through MM2's sources, I could not find a reference to this, we alway rely on the repository's prefix, not its name.
Question: Should we drop this? It makes things confusing and is basically noise since we do not use it anywhere.
== MD == Yes I think you should drop the column. It wound up not being useful for anything.
* Readable status of directories The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
== MD == Pre-bitflip content, which UMDL can see but the normal public can't yet. Are you no longer bitflipping? Then it doesn't matter.
* Changes while running Looking at the code, the UMDL seems to be very careful to handle changes on the FS while it is running. One hope I have is to speed up the UMDL run time, but I'm curious.
Question: Does anyone know if the FS changes often while the UMDL is actually running? Gaining speed of course does not mean being wreakless but I'm curious as to how often this situation occurs. IIRC, we trigger the UMDL via fedmsg now, right? So in theory, the FS shouldn't change too much under the UMDL's feet.
* The directory table So looking at the database and more precisely the directory table in that database, it seems we store all the directories of the tree, ie: /pub/alt/ /pub/alt/anaconda/ /pub/alt/bfo/ /pub/alt/bfo/gpxe-20120514 ... This makes me a little pondering. What is the interest of keeping the whole list of directories in the DB ? After all, as far as I understand, the UMDL finds the repo in the tree (repo being defined by the presence of a 'repodata' folder containing the repomd.xml or by the presence of a 'summary' file and an 'objects' folder). For these repo, we look for the most recent files, stores this info in the DB and later use it to check if the mirrors are up to date.
But do we need to checking that ``pub/fedora/linux`` exists when we later check that ``pub/fedora/linux/updates/testing/21/x86_64/`` exists and is up to date?
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in the crawler).
== MD == You need non-repo directories for ISOs at least; there was a time when we were able to mirror the entire Fedora static web content too; able only because MM tracked all directories, not just repository directories. MM1 also tried to be a "generic" mirror manager, not just a Fedora-specific mirror manager, so I intentionally tracked everything, not just Yum repos.
* Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree: * file * rsync * directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
== [file] was used early on for dev and testing. It's not interesting. [rsync] would be used when you don't have access to a master mirror (or very close replica). Perhaps the rpmfusion setup still needs this. I would have for testing Ubuntu, certainly. It shouldn't be needed for production when the content being mirrored out is managed by the same people operating mirrormanager, as is the Fedora case.
Looking forward hearing your thoughts about these points and questions, Thanks,
Pierre
On Fri, Jun 26, 2015 at 06:00:18PM +0000, Matt_Domsch@Dell.com wrote:
- Readable status of directories
The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
== MD == Pre-bitflip content, which UMDL can see but the normal public can't yet. Are you no longer bitflipping? Then it doesn't matter.
Ok, I see the use-case in the crawler, but in the UMDL, how did it work? The UMDL would not be allowed to read a given folder?
- Changes while running
Looking at the code, the UMDL seems to be very careful to handle changes on the FS while it is running. One hope I have is to speed up the UMDL run time, but I'm curious.
Question: Does anyone know if the FS changes often while the UMDL is actually running? Gaining speed of course does not mean being wreakless but I'm curious as to how often this situation occurs. IIRC, we trigger the UMDL via fedmsg now, right? So in theory, the FS shouldn't change too much under the UMDL's feet.
- The directory table
So looking at the database and more precisely the directory table in that database, it seems we store all the directories of the tree, ie: /pub/alt/ /pub/alt/anaconda/ /pub/alt/bfo/ /pub/alt/bfo/gpxe-20120514 ... This makes me a little pondering. What is the interest of keeping the whole list of directories in the DB ? After all, as far as I understand, the UMDL finds the repo in the tree (repo being defined by the presence of a 'repodata' folder containing the repomd.xml or by the presence of a 'summary' file and an 'objects' folder). For these repo, we look for the most recent files, stores this info in the DB and later use it to check if the mirrors are up to date.
But do we need to checking that ``pub/fedora/linux`` exists when we later check that ``pub/fedora/linux/updates/testing/21/x86_64/`` exists and is up to date?
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in the crawler).
== MD == You need non-repo directories for ISOs at least; there was a time when we were able to mirror the entire Fedora static web content too; able only because MM tracked all directories, not just repository directories. MM1 also tried to be a "generic" mirror manager, not just a Fedora-specific mirror manager, so I intentionally tracked everything, not just Yum repos.
Idea: what if we were tracking only the folders that have files in them, so for example http://dl.fedoraproject.org/pub/epel/5/ would not end-up in the database.
In addition, we could add a sort of blacklist to avoid storing http://dl.fedoraproject.org/pub/ just due to the presence of the DIRECTORY_SIZES.txt file
This would reduce the number of directories we store for the Atomic tree.
- Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree:
- file
- rsync
- directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
== [file] was used early on for dev and testing. It's not interesting. [rsync] would be used when you don't have access to a master mirror (or very close replica). Perhaps the rpmfusion setup still needs this. I would have for testing Ubuntu, certainly. It shouldn't be needed for production when the content being mirrored out is managed by the same people operating mirrormanager, as is the Fedora case.
Apparently RPMFusion does need this, so it needs to stay, the question becoming: Should we split the different UMDL types into different scripts? The idea being that allow easier optimization then. (Note: I'm having this idea now but since I did not looked at what/how we could optimize, it may end-up remaining in the same file)
Pierre
Dell Customer Communication -- Matt Domsch Senior Distinguished Engineer & Executive Director Dell | Software Group, Office of the CTO
-----Original Message----- From: Pierre-Yves Chibon [mailto:pingou@pingoured.fr] Sent: Monday, June 29, 2015 4:08 AM To: Domsch, Matt Cc: infrastructure@lists.fedoraproject.org Subject: Re: Thoughts and question about MM2's UMDL script
On Fri, Jun 26, 2015 at 06:00:18PM +0000, Matt_Domsch@Dell.com wrote:
- Readable status of directories
The Directory table has a 'readable' property, none of our directories is not readable.
Question is: what is the use-case for this boolean?
== MD == Pre-bitflip content, which UMDL can see but the normal public can't yet. Are you no longer bitflipping? Then it doesn't matter.
Ok, I see the use-case in the crawler, but in the UMDL, how did it work? The UMDL would not be allowed to read a given folder?
== MD == UMDL can read it, but the crawler can't. UMDL sets readable=False; crawler then doesn't delete the directory (or care if it can't read it) because it doesn't expect it to be readable. Otherwise, when readable=True but a given mirror doesn't have that content, crawler marks that host_category_directory for deletion.
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in the crawler).
== MD == You need non-repo directories for ISOs at least; there was a time when we were able to mirror the entire Fedora static web content too; able only because MM tracked all directories, not just repository directories. MM1 also tried to be a "generic" mirror manager, not just a Fedora-specific mirror manager, so I intentionally tracked everything, not just Yum repos.
Idea: what if we were tracking only the folders that have files in them, so for example http://dl.fedoraproject.org/pub/epel/5/ would not end-up in the database.
In addition, we could add a sort of blacklist to avoid storing http://dl.fedoraproject.org/pub/ just due to the presence of the DIRECTORY_SIZES.txt file
This would reduce the number of directories we store for the Atomic tree.
== MD == I didn't optimize for a few non-file-containing directories. You're welcome to if you see a need. But it's saving just a few entries out of hundreds/thousands.
- Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree:
- file
- rsync
- directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
== [file] was used early on for dev and testing. It's not interesting. [rsync] would be used when you don't have access to a master mirror (or very close replica). Perhaps the rpmfusion setup still needs this. I would have for testing Ubuntu, certainly. It shouldn't be needed for production when the content being mirrored out is managed by the same people operating mirrormanager, as is the Fedora case.
Apparently RPMFusion does need this, so it needs to stay, the question becoming: Should we split the different UMDL types into different scripts? The idea being that allow easier optimization then. (Note: I'm having this idea now but since I did not looked at what/how we could optimize, it may end-up remaining in the same file)
== MD == the parsing routing is pretty short; not worth a separate executable for.
On Mon, Jun 29, 2015 at 02:01:46PM +0000, Matt_Domsch@Dell.com wrote:
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in
the crawler).
== MD == You need non-repo directories for ISOs at least; there was a
time when we were able to mirror the entire Fedora static web content too; able only because MM tracked all directories, not just repository directories. MM1 also tried to be a "generic" mirror manager, not just a Fedora-specific mirror manager, so I intentionally tracked everything, not just Yum repos.
Idea: what if we were tracking only the folders that have files in them, so for example http://dl.fedoraproject.org/pub/epel/5/ would not end-up in the database.
In addition, we could add a sort of blacklist to avoid storing http://dl.fedoraproject.org/pub/ just due to the presence of the DIRECTORY_SIZES.txt file
This would reduce the number of directories we store for the Atomic tree.
== MD == I didn't optimize for a few non-file-containing directories. You're welcome to if you see a need. But it's saving just a few entries out of hundreds/thousands.
I got curious to see how it looks in reality, so I wrote a quick python script that goes through the entire tree and count the number of folders, files and folders with no files in them, this is how the results look like:
fedora 1814562 files found 4460 folders found 293 folders w/o files Ran in 11.111 min
epel 222697 files found 492 folders found 19 folders w/o files Ran in 1.400 min
alt 1309830 files found 13692 folders found 1614 folders w/o files Ran in 4.633 min
fedora-secondary 3774530 files found 5576 folders found 651 folders w/o files Ran in 26.701 min
archive 2705931 files found 3095 folders found 351 folders w/o files Ran in 22.042 min
Total time: 65.887 min
So it would save a few hundreds of entry in the directory table but it should still save some place in the host_category_dir table.
Also when seeing this, it feels to me that we should be more flexible about which part of the tree we run against, could even be a sub-part (ie: a specific secondary arch or so). I also would like to see if we can parallelize the browsing of the tree.
Pierre
On Fri, Jun 26, 2015 at 06:11:44PM +0200, Pierre-Yves Chibon wrote:
Yesterday and today I spent a little time going over the UDML script of MirrorManager2. Going through it, I ended up with few questions regarding it.
- Repository name
UMDL's code clearly says: # historically, Repository.name was a longer string with # product and category deliniations. But we were getting # unique constraint conflicts once we started introducing # repositories under repositories. And .name isn't used for # anything meaningful. So simply have it match dir.name, # which can't conflict. And quickly grepping through MM2's sources, I could not find a reference to this, we alway rely on the repository's prefix, not its name.
Question: Should we drop this? It makes things confusing and is basically noise since we do not use it anywhere.
It was a helpful column for fixing errors with the repos. But as the database is so huge everything we could drop should be dropped.
[...]
- The directory table
So looking at the database and more precisely the directory table in that database, it seems we store all the directories of the tree, ie: /pub/alt/ /pub/alt/anaconda/ /pub/alt/bfo/ /pub/alt/bfo/gpxe-20120514 ... This makes me a little pondering. What is the interest of keeping the whole list of directories in the DB ? After all, as far as I understand, the UMDL finds the repo in the tree (repo being defined by the presence of a 'repodata' folder containing the repomd.xml or by the presence of a 'summary' file and an 'objects' folder). For these repo, we look for the most recent files, stores this info in the DB and later use it to check if the mirrors are up to date.
But do we need to checking that ``pub/fedora/linux`` exists when we later check that ``pub/fedora/linux/updates/testing/21/x86_64/`` exists and is up to date?
I am under the impression currently that dropping un-necessary directories would save DB space (the directories being then linked in the host_category_dir table listing for each host, in each category which dir are present) as well as crawling time (both in the UMDL and in the crawler).
Again, dropping unnecessary information from the database sounds good. Although this one sounds a bit more complex as you always have to delete directories if subdirectories appear and add directories if subdirectories disappear.
- Non-directory based support in UDML.
So the UMDL script currently supports three ways of crawling the tree:
- file
- rsync
- directory
We, in Fedora, are only using the last one. I believe the `rsync` mode was added to support Ubuntu and the file mode is basically a simplified version of the directory mode, but that we do not use at at the moment.
I would like to propose that we drop support for rsync. I feel that it may be simpler and easier to create an UMDL and a crawler for each distro that would like to use MirrorManager than maintaining a one-script-fits-all UMDL that is in fact tested for only one of the scenario. That being said, if we ever have interest from Ubuntu, CentOS or any other communities, we should definitively look into making the UMDL and crawler as re-usable as possible for them, but keeping the distro-specific bits separated.
Like already mentioned, RPM Fusion uses the rsync mode as the master mirror is 'far' away from the MirrorManager installation. It is still using MM1 on CentOS 5 and currently I am not immediately planing on upgrading to MM2. So it could be removed and I should be able to write the necessary umdl rsync crawler once I need it.
Another thought about umdl I had concerns the file mode. We have for the categories 'Fedora EPEL' and 'Fedora Linux' files called 'fullfilelist'. Maybe that would be an option for umdl to use to reduce I/O on the NFS mounts. Only actually reading the files and metadata from NFS if it is necessary. Just one of those ideas.
Adrian
On Fri, Jun 26, 2015 at 08:45:18PM +0200, Adrian Reber wrote:
Another thought about umdl I had concerns the file mode. We have for the categories 'Fedora EPEL' and 'Fedora Linux' files called 'fullfilelist'. Maybe that would be an option for umdl to use to reduce I/O on the NFS mounts. Only actually reading the files and metadata from NFS if it is necessary. Just one of those ideas.
I like this! It would also speed up the idea which I had, this would then be a new type `fullfilelist` :)
Pierre
infrastructure@lists.fedoraproject.org
-
Adrian Reber -
Kevin Fenzi -
Matt Domsch -
Pierre-Yves Chibon -
Stephen John Smoogen