Hi,
I've started to work in order to allow debian provisioning with cobbler. I send attached two patches.
The first on is actually not strictly related to debian, and it's a reorganization of the archs management, to simplify it and ease smooth addition of other distros: debian calls amd64 to x86_64, and the idea is to keep the RedHat names as canonical ones.
The second one does actually allow to import the content of a debian CD/DVD and create the corresponding distro. It also creates the profile breeded as debian, although it might not be usable yet.
There are a couple of open questions that might affect other parts of cobbler or where just ideas are required.
- the kickstarts for debian. They are called "preseed", and are by far much more complex than a standard kickstart. Although not strictly required, it might be better to have new names for both ks files and directories, but I think that they can be easyly added to PXE boot options. To make things worst, ubuntu does allow both the standard debian preseed and kickstart files.
- the package repositories. Debian follows a quite different approach, and a single tree can hold many versions and different architectures. Allowing a cobbler distro to have multiple architectures might reduce the disk usage and simplify the repository updating. Due to this fact, a single debian media can install multiple archs, complexing a little bit to import the media.
These are the only two points I see on the close horizon, but there are probably more. It anyone has concerns or ideas, they are welcome. And if anyone can actually test the deployment (when it matures a little bit), much better.
Javier Palacios
Javier Palacios wrote:
Hi,
I've started to work in order to allow debian provisioning with cobbler. I send attached two patches.
The first on is actually not strictly related to debian, and it's a reorganization of the archs management, to simplify it and ease smooth addition of other distros: debian calls amd64 to x86_64, and the idea is to keep the RedHat names as canonical ones.
The second one does actually allow to import the content of a debian CD/DVD and create the corresponding distro. It also creates the profile breeded as debian, although it might not be usable yet.
There are a couple of open questions that might affect other parts of cobbler or where just ideas are required.
- the kickstarts for debian. They are called "preseed", and are by far
much more complex than a standard kickstart. Although not strictly required, it might be better to have new names for both ks files and directories, but I think that they can be easyly added to PXE boot options. To make things worst, ubuntu does allow both the standard debian preseed and kickstart files.
- the package repositories. Debian follows a quite different approach,
and a single tree can hold many versions and different architectures. Allowing a cobbler distro to have multiple architectures might reduce the disk usage and simplify the repository updating. Due to this fact, a single debian media can install multiple archs, complexing a little bit to import the media.
These are the only two points I see on the close horizon, but there are probably more. It anyone has concerns or ideas, they are welcome. And if anyone can actually test the deployment (when it matures a little bit), much better.
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
First off, thanks very much for the patch. The first one is going to need a minor bit of tweaking to put the constants back to the way Cobbler used them, though the generalized code cleanup is a good in priciple -- I do not want the constants renumbered or mapped to provide aliases for one another. I'll explain why. I had thought you were wanting to rewrite this more than what you had submitted.
Ok, so the reason we have consolidated on x86 and x86_64 (not i386 or i686) is to ensure that we don't break things unexpectedly with respect to yum. There i386 is incorrect there as it will not pull in i686 kernel packages, so there we need i686, which is why the internal represenation is stored as x86. And since we sometimes compare the arches of distros with the arches of repo objects they should be stored the same way.
We would also never have "bsd_x86_64" as an arch, so remove that suggestion from the comments so that does not confuse someone reading the code. The way we do that is with the "--breed" parameter, which is seperate from arch, we don't combine them. This is the way we do SuSE and debian already today:
cobbler distro add --name=foo --arch=x86_64 ... # implies redhat based cobbler distro add --name=foo --breed=debian --arch=x86_64 ... cobbler distro add --name=foo --breed=suse --arch=x86_64 ...
I am also confused by:
+ if os.path.realpath(fullname) == os.path.realpath(dirname): + print _("- found self-refering link, probably debian based distro") + self.breed_guess = "debian" + continue
This seems to be a kind of strange way to detect that a tree is Debian based. A better route is probably to have two seperate importer classes, which I believe someone might have been working on, and require when importing non-redhat distributions that we specify that is what they are.
For instance, add a new "--breed" parameter to import.
This may take some refactoring of the import code but would be better than further complicating it, as already it has to deal with a lot of variance between different EL and Fedora distributions. Having that become /more/ complex is really not a good option.
So ideally I think what we want is to see action_import.py become a switchboard, and then based on the value of --breed given to the import call, choose the right importer_*.py out of "modules" -- that way we can arbitrarily support other distributions without having to keep adding the code to the same importer, which will likely become more complicated to follow as we add more and more code to it.
--Michael
I'll try to reply both mails within this one.
Regarding the multiple architectures repos and DVD, i also thought about just don't support them. But I wanted to check if they may fit in some manner. As I understand, a single distro can have multiple profiles (with different ks), so it might be more or less ready to support multiple architectures (a different pxe config + ks).
The default preseed is on the close future list, though I want to have a closer look into how kickstarts are managed. And the repo managemen will hopefully be easy, as the only tool I know to mirror debian repositories is already written in python (debmirror).
First off, thanks very much for the patch. The first one is going to need a minor bit of tweaking to put the constants back to the way Cobbler used them, though the generalized code cleanup is a good in priciple -- I do not want the constants renumbered or mapped to provide aliases for one another. I'll explain why. I had thought you were wanting to rewrite this more than what you had submitted.
Both the "canonical" and the map lists were initially internal variables for import. But I saw that the "wrong" names were used in other places (specially on pxegen, which brokes the consistency, I decided to move into utils package, to made it widely available. The map is actually intended only to map architecture names from files and directories into the canonical names, so it should be internal to import, but I moved also into utils because it might be useful for serializers in order to ensure the arch names in the files become the canonical ones. In the mid-long term, the map should go again into import, unless other cobbler part need non user supplied architectures.
Ok, so the reason we have consolidated on x86 and x86_64 (not i386 or i686) is to ensure that we don't break things unexpectedly with respect to yum. There i386 is incorrect there as it will not pull in i686 kernel packages, so there we need i686, which is why the internal represenation is stored as x86. And since we sometimes compare the arches of distros with the arches of repo objects they should be stored the same way.
Should I understand that x86 is the canonical name for i386? The import code did transform x86 into i386, so I guessed the latter is the good one. But your paragraph points that pxegen does the right transformation and that the real bug was in action_import. If that is the case, let me correct the patch before even thinking of apply them, as it comes from a misunderstanding.
cobbler distro add --name=foo --arch=x86_64 ... # implies redhat based cobbler distro add --name=foo --breed=debian --arch=x86_64 ... cobbler distro add --name=foo --breed=suse --arch=x86_64 ...
I am also confused by:
if os.path.realpath(fullname) == os.path.realpath(dirname):
print _("- found self-refering link, probably debian
based distro")
self.breed_guess = "debian"
continue
This seems to be a kind of strange way to detect that a tree is Debian based. A better route is probably to have two seperate importer classes, which I believe someone might have been working on, and require when importing non-redhat distributions that we specify that is what they are.
For instance, add a new "--breed" parameter to import.
I didn't want to modify the CLI usage for importing. And I wasn't aware of the breed argument for distro importing. I also believe that the easier and the better is to add a breed parameter also to import, so I'll do that. Just a question, if some inconsistency is found from command line to real tree (as specify a redhat tree and founding a "debian" symlink), should I just raise an exception/error or start a user dialog ?
This may take some refactoring of the import code but would be better than further complicating it, as already it has to deal with a lot of variance between different EL and Fedora distributions. Having that become /more/ complex is really not a good option.
So ideally I think what we want is to see action_import.py become a switchboard, and then based on the value of --breed given to the import call, choose the right importer_*.py out of "modules" -- that way we can arbitrarily support other distributions without having to keep adding the code to the same importer, which will likely become more complicated to follow as we add more and more code to it.
I'll take into account the modules approach, but after going deeper into the code, now I think it is not so much redhat-centric as initially appears. It's more about the names used than about the underlying logic.
Javier Palacios
On Fri, Aug 29, 2008 at 10:23:45AM +0200, Javier Palacios wrote:
Ok, so the reason we have consolidated on x86 and x86_64 (not i386 or i686) is to ensure that we don't break things unexpectedly with respect to yum. There i386 is incorrect there as it will not pull in i686 kernel packages, so there we need i686, which is why the internal represenation is stored as x86. And since we sometimes compare the arches of distros with the arches of repo objects they should be stored the same way.
Should I understand that x86 is the canonical name for i386? The import code did transform x86 into i386, so I guessed the latter is the good one. But your paragraph points that pxegen does the right transformation and that the real bug was in action_import. If that is the case, let me correct the patch before even thinking of apply them, as it comes from a misunderstanding.
In Fedora/RHEL world i386 means more or less ia32. In kernel/rpm speech i686 is a subarchitecture of i386.
yum/anaconda on i386 do query the system to decide on whether to pull in i586 vs i686 vs athlon packages for kernels/glibc (well it did, most of the athlon stuff has been consolidated into i686 by now, but eleder RHEL like RHEL3 still have this distinction).
Axel Thimm wrote:
On Fri, Aug 29, 2008 at 10:23:45AM +0200, Javier Palacios wrote:
Ok, so the reason we have consolidated on x86 and x86_64 (not i386 or i686) is to ensure that we don't break things unexpectedly with respect to yum. There i386 is incorrect there as it will not pull in i686 kernel packages, so there we need i686, which is why the internal represenation is stored as x86. And since we sometimes compare the arches of distros with the arches of repo objects they should be stored the same way.
Should I understand that x86 is the canonical name for i386?
It's the name we are going to use to describe 32 bit systems to avoid the baggage of debating whether "i386" referers to i386 compatible instructions or can include i686, which is pretty much omnipresent now.
The import code did transform x86 into i386, so I guessed the latter is the good one. But your paragraph points that pxegen does the right transformation and that the real bug was in action_import. If that is the case, let me correct the patch before even thinking of apply them, as it comes from a misunderstanding.
In Fedora/RHEL world i386 means more or less ia32. In kernel/rpm speech i686 is a subarchitecture of i386.
yum/anaconda on i386 do query the system to decide on whether to pull in i586 vs i686 vs athlon packages for kernels/glibc (well it did, most of the athlon stuff has been consolidated into i686 by now, but eleder RHEL like RHEL3 still have this distinction).
Yum's reposync seemed to want i686 passed in for the arch, if not running on that arch, to grab the kernel packages, FWIW. It did not like "i386" in that case.
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Should I understand that x86 is the canonical name for i386?
It's the name we are going to use to describe 32 bit systems to avoid the baggage of debating whether "i386" referers to i386 compatible instructions or can include i686, which is pretty much omnipresent now.
Once solved my misunderstanding, I submit the patch again, with the right architecture mapping. This time I've preferred not to touch anything not directly related to import, and I've enlarged the patch a little bit. Now it performs a complete import, including a minimal sample.seed file. It still does not write right append options for pxe in order to actually find the location of the preseed file.
Javier Palacios
Javier Palacios wrote:
Should I understand that x86 is the canonical name for i386?
It's the name we are going to use to describe 32 bit systems to avoid the baggage of debating whether "i386" referers to i386 compatible instructions or can include i686, which is pretty much omnipresent now.
Once solved my misunderstanding, I submit the patch again, with the right architecture mapping. This time I've preferred not to touch anything not directly related to import, and I've enlarged the patch a little bit. Now it performs a complete import, including a minimal sample.seed file. It still does not write right append options for pxe in order to actually find the location of the preseed file.
This is what --breed does in cobbler. If you set breed, the values given for set_kickstart() automatically are mapped such that they can be found by the Debian installer.
Let me know when it's complete and I'll take a look at it.
--Michael
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
On Tue, Sep 2, 2008 at 4:01 PM, Michael DeHaan mdehaan@redhat.com wrote:
it performs a complete import, including a minimal sample.seed file. It still does not write right append options for pxe in order to actually find the location of the preseed file.
This is what --breed does in cobbler. If you set breed, the values given for set_kickstart() automatically are mapped such that they can be found by the Debian installer.
Let me know when it's complete and I'll take a look at it.
Well, actually it is completed, at least in the sense that ticket #112 could be marked as completed (but untested). Both kickstart_finder and set_variance methods are already patched to supply a sensible debian preseed (instead of a ks file) in case no kickstart file is specified on command line.
What is missing is not actually related to import, but to create proper pxe boot entries (debian does not understand ks=...), and allow the usage of templates for preseed files. As the kicstart/preseed might be considered "optional" features, I think it is better to focus now in managing the debian repo to keep packages updated.
Javier Palacios
Javier Palacios wrote:
On Tue, Sep 2, 2008 at 4:01 PM, Michael DeHaan mdehaan@redhat.com wrote:
it performs a complete import, including a minimal sample.seed file. It still does not write right append options for pxe in order to actually find the location of the preseed file.
This is what --breed does in cobbler. If you set breed, the values given for set_kickstart() automatically are mapped such that they can be found by the Debian installer.
Let me know when it's complete and I'll take a look at it.
Well, actually it is completed, at least in the sense that ticket #112 could be marked as completed (but untested). Both kickstart_finder and set_variance methods are already patched to supply a sensible debian preseed (instead of a ks file) in case no kickstart file is specified on command line.
What is missing is not actually related to import, but to create proper pxe boot entries (debian does not understand ks=...), and allow the usage of templates for preseed files. As the kicstart/preseed might be considered "optional" features, I think it is better to focus now in managing the debian repo to keep packages updated.
As I mentioned earlier, setting the breed makes sure the kernel arguments are correct automatically. It needs to do this and is already supported.
Fix this and I'll see about applying it.
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
On Wed, Sep 3, 2008 at 3:09 PM, Michael DeHaan mdehaan@redhat.com wrote:
As I mentioned earlier, setting the breed makes sure the kernel arguments are correct automatically. It needs to do this and is already supported.
Fix this and I'll see about applying it.
I maybe didn't explain that well. The patch actually set the breed as well as the version, by calling methods set_breed and set_os_version.
Javier Palacios
Javier Palacios wrote:
On Wed, Sep 3, 2008 at 3:09 PM, Michael DeHaan mdehaan@redhat.com wrote:
As I mentioned earlier, setting the breed makes sure the kernel arguments are correct automatically. It needs to do this and is already supported.
Fix this and I'll see about applying it.
I maybe didn't explain that well. The patch actually set the breed as well as the version, by calling methods set_breed and set_os_version.
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Then the kernel arguments would be correct. I'm not sure why you are stating they would not be?
Then the kernel arguments would be correct. I'm not sure why you are stating they would not be?
I did refer to the kickstart ones, but as I didn't actually create any dhcp/pxe entry I was just supposing. Anaconda understand ks=url_or_path, but the debian installer expects url=url_or_path. As I see that there was no legacy.seed or sample.seed (as do exists for real kickstarts), I didn't expect other debian specifics implemented.
Javier Palacios
Javier Palacios wrote:
Then the kernel arguments would be correct. I'm not sure why you are stating they would not be?
I did refer to the kickstart ones, but as I didn't actually create any dhcp/pxe entry I was just supposing. Anaconda understand ks=url_or_path, but the debian installer expects url=url_or_path. As I see that there was no legacy.seed or sample.seed (as do exists for real kickstarts), I didn't expect other debian specifics implemented.
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Yes, they are already implemented, as is support for the SuSE AutoYAST variations. Grep for "breed" in the code and you will see where this is done.
I will take a look at this in the next few days.
Thanks!
--Michael
Javier Palacios wrote:
Should I understand that x86 is the canonical name for i386?
It's the name we are going to use to describe 32 bit systems to avoid the baggage of debating whether "i386" referers to i386 compatible instructions or can include i686, which is pretty much omnipresent now.
Once solved my misunderstanding, I submit the patch again, with the right architecture mapping. This time I've preferred not to touch anything not directly related to import, and I've enlarged the patch a little bit. Now it performs a complete import, including a minimal sample.seed file. It still does not write right append options for pxe in order to actually find the location of the preseed file.
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Ok, reviewing this again, there are a few problems. Needless to say I'm highly interested in this feature, though I can't accept it if it causes problems for the existing support of some of the ways trees are handled today. Import has been a problem in the past and testing it takes a lot of time, so this is why import changes need to be minimal and/or kept to other modules/commands.
Looking at it again, the archmap stuff should really not be in there. Basically what we need here is the bare minimum patch to add "cobbler import" ability to scan Debian, without changing or cleaning up any of the existing stuff.
For reference, here's one of the import use cases that I found broken:
(A) Imported distros now have the "x86" suffix instead of i386, even though the arch is still correctly set to i386. For consistency they should be left as "i386" as they are in the trees themselves when the import is performed.
(B) When attempting to correct this (removing the archmap code), the following command now fails to work:
make devinstall && make eraseconfig && cobbler import --name=RHEL3U3 --mirror=/mnt/engarchive2/released/RHEL-3/U3/AS/ --available-as=http://foo
(eraseconfig is a new target that just creates a clean cobbler install)
The above import takes a tree containing:
[root@localhost cobbler]# ls /mnt/engarchive2/released/RHEL-3/U3/AS/ i386 ia64 ppc s390 s390x x86_64
And imports all of the distributions, creating objects for each. For instance, it will find the appropriate tree distros:
[root@localhost cobbler]# ls /mnt/engarchive2/released/RHEL-3/U3/AS/i386/tree autorun RELEASE-NOTES-es.html RELEASE-NOTES-U2-it.html dosutils RELEASE-NOTES-fr.html RELEASE-NOTES-U2-ja.html EULA RELEASE-NOTES-it.html RELEASE-NOTES-U2-ko.html GPL RELEASE-NOTES-ja.html RELEASE-NOTES-U2-pt_BR.html images RELEASE-NOTES-ko.html RELEASE-NOTES-U2-zh_CN.html isolinux RELEASE-NOTES-pt_BR.html RELEASE-NOTES-U2-zh_TW.html README-Accessibility RELEASE-NOTES-U1-de.html RELEASE-NOTES-U3-de.html README-de.html RELEASE-NOTES-U1-en RELEASE-NOTES-U3-en README-en RELEASE-NOTES-U1-en.html RELEASE-NOTES-U3-en.html README-en.html RELEASE-NOTES-U1-es.html RELEASE-NOTES-U3-es.html README-es.html RELEASE-NOTES-U1-fr.html RELEASE-NOTES-U3-fr.html README-fr.html RELEASE-NOTES-U1-it.html RELEASE-NOTES-U3-it.html README-it.html RELEASE-NOTES-U1-ja.html RELEASE-NOTES-U3-ja.html README-ja.html RELEASE-NOTES-U1-ko.html RELEASE-NOTES-U3-ko.html README-ko.html RELEASE-NOTES-U1-pt_BR.html RELEASE-NOTES-U3-pt_BR.html README-pt_BR.html RELEASE-NOTES-U1-zh_CN.html RELEASE-NOTES-U3-zh_CN.html README-zh_CN.html RELEASE-NOTES-U1-zh_TW.html RELEASE-NOTES-U3-zh_TW.html README-zh_TW.html RELEASE-NOTES-U2-de.html RELEASE-NOTES-zh_CN.html RedHat RELEASE-NOTES-U2-en RELEASE-NOTES-zh_TW.html RELEASE-NOTES-de.html RELEASE-NOTES-U2-en.html RPM-GPG-KEY RELEASE-NOTES-en RELEASE-NOTES-U2-es.html RPM-GPG-KEY-beta RELEASE-NOTES-en.html RELEASE-NOTES-U2-fr.html SRPMS
Since not all arches are supported this should create...
i386 ia64 s390x x86_64
The new archwalker in this patch seems to not identify an arch for "ls /mnt/engarchive2/released/RHEL-3/U3/" which means it stopped too early, which is why I am suggesting that Debian support be added without trying to refactor/cleanup the code -- basically we just need to keep it simple. Try to get the bare minimum to add the import in there, so that the other import cases (including older RHEL) still work and resist the urge to refactor things as they may not work as expected once changed.
I don't really care if the import code is ugly or not -- it's going to be ugly regardless -- the main thing is that it works (this is glue code after all) and the various use cases for older distros and importing multiple distributions at the same time (say the distro trees all live on NFS on a filer somewhere) continue to work.
Again, I'm really really happy to see the interest in getting "cobbler import" to work on Debian, I just need to make sure that these changes are right for the existing Cobbler userbase as well so things keep working for the other distros.
Thanks!
--Michael
Just a few fast remarks
Looking at it again, the archmap stuff should really not be in there. Basically what we need here is the bare minimum patch to add "cobbler import" ability to scan Debian, without changing or cleaning up any of the existing stuff.
The archmap stuff is there just to replace existing changes in arch names coming from command line or from the imported trees. It's no problem to leave the original code there (although I obviously believe that the map is cleaner, besides the ugly trick of the empty string value).
For reference, here's one of the import use cases that I found broken:
make devinstall && make eraseconfig && cobbler import --name=RHEL3U3 --mirror=/mnt/engarchive2/released/RHEL-3/U3/AS/ --available-as=http://foo
Is it possible to get a copy of that tree ? When I run `make test`, everything goes right, although the included repos are quite limited. [Actually I need to remove acl related things. Do I need some extra package, or might that code not be commited]
(eraseconfig is a new target that just creates a clean cobbler install)
Regarding that target, I believe there is a bug in the Makefile. It cares about /var/lib/cobbler/distro* and similar things, but I think what it searchs for is actually /var/lib/cobbler/config/distro*
Javier Palacios
Javier Palacios wrote:
Just a few fast remarks
Looking at it again, the archmap stuff should really not be in there. Basically what we need here is the bare minimum patch to add "cobbler import" ability to scan Debian, without changing or cleaning up any of the existing stuff.
The archmap stuff is there just to replace existing changes in arch names coming from command line or from the imported trees. It's no problem to leave the original code there (although I obviously believe that the map is cleaner, besides the ugly trick of the empty string value).
It did a bit more than that -- it changed behavior from the way cobbler used to work -- "Fedora i386" imported as "Fedora x86", for instance, even if the arch was set correctly to i386 to correspond with the tree.
I would suggest doing the bare minimum to support Debian imports rather than trying to fix this -- it requires much less debugging and testing when we start with a base that's known to work and then just apply a minimal patch.
For reference, here's one of the import use cases that I found broken:
make devinstall && make eraseconfig && cobbler import --name=RHEL3U3 --mirror=/mnt/engarchive2/released/RHEL-3/U3/AS/ --available-as=http://foo
Is it possible to get a copy of that tree ? When I run `make test`, everything goes right, although the included repos are quite limited. [Actually I need to remove acl related things. Do I need some extra package, or might that code not be commited]
This is not part of "make test", but the result of running the import command above. That tree is pretty huge, but you can simulate it by having several DVD's exploded in the same root. "make test" does not do large imports like that because not everyone will have access to the same tree.
(eraseconfig is a new target that just creates a clean cobbler install)
Regarding that target, I believe there is a bug in the Makefile. It cares about /var/lib/cobbler/distro* and similar things, but I think what it searchs for is actually /var/lib/cobbler/config/distro*
Not a bug. The makefile is set to ignore commands that fail and try two possibilities.
serializer_catalog (the new default) uses /var/lib/cobbler/config/distros.d/*
serializer_yaml (the old default) uses /var/lib/cobbler/distros
This is configured in modules.conf, though most people will never need to change the setting.
It cleans them both out so there's no copy of either.
--Michael
Hi,
I send attached a new version of the patch. It is able to import debian and ubuntu distros, and I've actually tested successfully importing a debian multiarch DVD, an ubuntu alternate CD, and local mirrors of Fedora 9 (32 & 64 bit) and redhat 4.6.
If trying to import multiple distros at the same time, it will not exception, but will not detect the flavour, and will assign a wrong value for kickstart url.
This will also happens if the imported tree does not exactly fit that from real media. As I tried to explain in my previous mail, current import decides that the tree base is two directories over the location of kernel and initrd. This is true for redhat based distros just because the ones at isolinux directory are rejected. But this is not true for debian/ubuntu and probably for other distros.
It should be decided if it is desired "cobbler import" to work on single media of able to work with multidistro trees. And as far as I see the multi-tree approach will difficult extension to non-redhat distros, and will not help to cleaning the import code.
Javier Palacios
Javier Palacios wrote:
Hi,
I send attached a new version of the patch. It is able to import debian and ubuntu distros, and I've actually tested successfully importing a debian multiarch DVD, an ubuntu alternate CD, and local mirrors of Fedora 9 (32 & 64 bit) and redhat 4.6.
If trying to import multiple distros at the same time, it will not exception, but will not detect the flavour, and will assign a wrong value for kickstart url.
This will also happens if the imported tree does not exactly fit that from real media. As I tried to explain in my previous mail, current import decides that the tree base is two directories over the location of kernel and initrd. This is true for redhat based distros just because the ones at isolinux directory are rejected. But this is not true for debian/ubuntu and probably for other distros.
It should be decided if it is desired "cobbler import" to work on single media of able to work with multidistro trees. And as far as I see the multi-tree approach will difficult extension to non-redhat distros, and will not help to cleaning the import code.
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Thanks very much!
I'll check this over after I finish testing the cloning items. Patch looks fine, though I need to run it through some of the EL/Fedora use cases too for testing.
Multi-import is probably not critical if we document the limitation, though if we can get it doing that, it would probably be nice to have. I'm not that worried about it and we can explain that on the Wiki and in the manpage if need be.
--Michael
I'll check this over after I finish testing the cloning items. Patch looks fine, though I need to run it through some of the EL/Fedora use cases too for testing.
I expect your comments, though I will start looking to opensuse media
Multi-import is probably not critical if we document the limitation, though if we can get it doing that, it would probably be nice to have. I'm not that worried about it and we can explain that on the Wiki and in the manpage if need be.
The multi-import will very likely be recovered, although it might require a command line option to trigger the breed specific layouts. Anyway, I'm trying to figure how to detect the media root from the isolinux.cfg file, which is usually also present on web mirrors.
Javier Palacios
Javier Palacios wrote:
I'll check this over after I finish testing the cloning items. Patch looks fine, though I need to run it through some of the EL/Fedora use cases too for testing.
I expect your comments, though I will start looking to opensuse media
Still testing now, Fedora and RHEL looks great, off to try some ISOs of other things.
Very cool to hear you're looking at SuSE too. A fair amount of folks have asked for that and it will be nice to have.
Multi-import is probably not critical if we document the limitation, though if we can get it doing that, it would probably be nice to have. I'm not that worried about it and we can explain that on the Wiki and in the manpage if need be.
The multi-import will very likely be recovered, although it might require a command line option to trigger the breed specific layouts. Anyway, I'm trying to figure how to detect the media root from the isolinux.cfg file, which is usually also present on web mirrors.
Neat! Yeah, using more data if available would be great...
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Javier Palacios wrote:
I'll check this over after I finish testing the cloning items. Patch looks fine, though I need to run it through some of the EL/Fedora use cases too for testing.
I expect your comments, though I will start looking to opensuse media
Multi-import is probably not critical if we document the limitation, though if we can get it doing that, it would probably be nice to have. I'm not that worried about it and we can explain that on the Wiki and in the manpage if need be.
The multi-import will very likely be recovered, although it might require a command line option to trigger the breed specific layouts. Anyway, I'm trying to figure how to detect the media root from the isolinux.cfg file, which is usually also present on web mirrors.
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Seems to work absolutely great on all classes of imports. Thanks very very much.
One small snag I found is that when using sample.seed from the import against debian-40r4a-etchnhalf-i386-netinst.iso I get the following distro and profile as expected:
[root@localhost cobbler]# cobbler distro report distro : debian-i386 breed : debian os version : debianEtch architecture : i386 initrd : /var/www/cobbler/ks_mirror/debian/install.386/initrd.gz kernel : /var/www/cobbler/ks_mirror/debian/install.386/vmlinuz kernel options : {} ks metadata : {'tree': 'http://@@http_server@@/cblr/links/debian-i386%27%7D mgmt classes : [] owners : ['admin'] post kernel options : {}
[root@localhost cobbler]# cobbler profile report profile : debian-i386 distro : debian-i386 enable menu : True dhcp tag : default kernel options : {} post kernel options : {} kickstart : /etc/cobbler/sample.seed ks metadata : {} mgmt classes : [] owners : ['admin'] repos : [] server : <<inherit>> virt bridge : xenbr0 virt cpus : 1 virt file size : 5 virt path : virt ram : 512 virt type : qemu
Except that the Debian installer reads "Your installation CD-ROM could not be mounted. This probably means that the CD-ROM was not in the drive. If so you can insert it and try again. Try again to mount hte CD-ROM? Yes/No"
Looking at the PXE config I have the following which looks correct
LABEL debian-i386 kernel /images/debian-i386/vmlinuz MENU LABEL debian-i386 append initrd=/images/debian-i386/initrd.gz interface=eth0 lang= kssendmac syslog=172.16.57.21:25150 text auto=true url=http://172.16.57.21/cblr/svc/op/ks/profile/debian-i386
And if I visit that URL I can verify it serves up the following preseed:
#platform=x86, AMD64, or Intel EM64T # System authorization information
# System bootloader configuration
# Partition clearing information
# Use text mode install
# Firewall configuration
# Run the Setup Agent on first boot
# System keyboard d-i console-setup/dont_ask_layout note d-i console-setup/layoutcode string us d-i console-setup/layout select U.S. English # System language
# Use network installation #d-i mirror/http/hostname string mirror.domain #d-i mirror/http/directory string /debian_directory # If any cobbler repo definitions were referenced in the kickstart profile, include them here.
# Network information
# Reboot after installation finish-install finish-install/reboot_in_progress note
#Root password d-i passwd/root-password-crypted password $1$mF86/UHC$WvcIcX2t6crBz2onWxyac. user-setup-udeb passwd/root-login boolean true user-setup-udeb passwd/make-user boolean false # SELinux configuration
# Do not configure the X Window System
# System timezone clock-setup clock-setup/utc boolean false tzsetup-udeb time/zone select America/New_York # Install OS instead of upgrade
# Clear the Master Boot Record
So what it looks like to me is that the preseed file just needs some work to make this fully automated, or else I should not be using the netinst image with the import?
Either way, let me know. I think this would be really awesome if we could use this to replace FAI and also get Cobbler packaged in the Debian trees.
It looks like to me the only problem is that the variable $tree or $server (or whatever else is needed ... see "cobbler profile dumpvars --name=foo" to see what you can choose from aren't referenced in the sample.seed file, so it's not yet fully configured to use the cobbler server as a mirror.
Can you add this?
Anyway, I've applied everything to devel and otherwise looks awesome. Thanks very much for working on this!
--Michael
Except that the Debian installer reads "Your installation CD-ROM could not be mounted. This probably means that the CD-ROM was not in the drive. If so you can insert it and try again. Try again to mount hte CD-ROM? Yes/No"
[...]
So what it looks like to me is that the preseed file just needs some work to make this fully automated, or else I should not be using the netinst image with the import?
Either way, let me know. I think this would be really awesome if we could use this to replace FAI and also get Cobbler packaged in the Debian trees.
The sample.seed will produce a partly automated install (specifically, should not ask for language and keyboard), but the message about missing CD is very likely due to absence of network enabled initrd. If possible, try an ubuntu import, as the initrd is probably more versatile. In the case of debian, the netinstall CD will not be imported because it has only a isolinux directory, but as far as I remember there is a "heavy" network install CD that might work. Anyway, I'll investigate this.
Anyway, I've applied everything to devel and otherwise looks awesome. Thanks very much for working on this!
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Javier Palacios
Javier Palacios wrote:
Except that the Debian installer reads "Your installation CD-ROM could not be mounted. This probably means that the CD-ROM was not in the drive. If so you can insert it and try again. Try again to mount hte CD-ROM? Yes/No"
[...]
So what it looks like to me is that the preseed file just needs some work to make this fully automated, or else I should not be using the netinst image with the import?
Either way, let me know. I think this would be really awesome if we could use this to replace FAI and also get Cobbler packaged in the Debian trees.
The sample.seed will produce a partly automated install (specifically, should not ask for language and keyboard), but the message about missing CD is very likely due to absence of network enabled initrd. If possible, try an ubuntu import, as the initrd is probably more versatile. In the case of debian, the netinstall CD will not be imported because it has only a isolinux directory, but as far as I remember there is a "heavy" network install CD that might work. Anyway, I'll investigate this.
Could be, but I think there's more to it ... how does it know the content is in /var/www/cobbler when it is not referenced in either the kernel options or the preseed?
I think that's the problem.
At least with Anaconda behavior, the same kind of thing pops up if you don't specify the install source.
Can you check the preseed in git to see if it looks right ot you? I possibly remember something being added in a later patch.
Getting it to full auto would also be nice, as then we can trumpet ourselves as a nice replacement for FAI. Getting full auto is of course the "FA" in FAI, so we'd want it to do that straight off of import.
Anyway, I've applied everything to devel and otherwise looks awesome. Thanks very much for working on this!
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Excellent.
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Actually recover the multi-import has been much easier than I expected. The patch attached works, at least for me.
I can import three distros as cobbler import --mirror=/media/distros/rhas/4.6 --name=rhas cobbler import --path=/media/distros/Fedora9/i386/ --name=f9_32 cobbler import --mirror=/media/distros/Fedora9/x86_64 --name=f9_64
or issue a single command cobbler import --mirror=/media/distros --name=allinone --breed=redhat
and all of them work.
Regarding the patch itself, a couple of comments. I've moved distro_adder method close to the distro add related method, but there is no change there. I have also modified item_distro.py, to remove breed and arch defaults. It was difficult to know why (and where) was the distro flavor set without os version until I realize that the redhat flavor was given on instantiation. It might be safer not to move that change into HEAD, but at least for devel branch I believe it is a good idea to have no such hardcoded defaults. The remaining changes are mostly intended as a first step to the real refactor, to make easier understand the changes than submitting the patch in the final form
Javier Palacios
Javier Palacios wrote:
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Actually recover the multi-import has been much easier than I expected. The patch attached works, at least for me.
I can import three distros as cobbler import --mirror=/media/distros/rhas/4.6 --name=rhas cobbler import --path=/media/distros/Fedora9/i386/ --name=f9_32 cobbler import --mirror=/media/distros/Fedora9/x86_64 --name=f9_64
or issue a single command cobbler import --mirror=/media/distros --name=allinone --breed=redhat
and all of them work.
Regarding the patch itself, a couple of comments. I've moved distro_adder method close to the distro add related method, but there is no change there. I have also modified item_distro.py, to remove breed and arch defaults. It was difficult to know why (and where) was the distro flavor set without os version until I realize that the redhat flavor was given on instantiation. It might be safer not to move that change into HEAD, but at least for devel branch I believe it is a good idea to have no such hardcoded defaults. The remaining changes are mostly intended as a first step to the real refactor, to make easier understand the changes than submitting the patch in the final form
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Ok, I will look at this shortly.
Can you comment on the missing "$tree" in the seed file and how we can get the preseed and/or kernel options to reference the install source? This was from Friday's comment.
--Michael
Can you comment on the missing "$tree" in the seed file and how we can get the preseed and/or kernel options to reference the install source? This was from Friday's comment.
The value I get out from cobbler profile dumpvars --name=debian-xxx seems sensible, although the actual symlink on filesystem is wrong and points just to top ks_mirror. It might be because the code I'm using now is a bit evolved respect to current git contents.
Anyway, and despite the fact that the kernel/initrd from cd might not be prepared for network install, there is plenty of work required for even partly automated installs. For example, the simple url --url=$tree statement in the kickstart sample is far away from debian's counterpart. FIrst of all, in the preseed the url is actually splitted in three different parts: the protocol, the host and the directory. Right now I cannot perform any test at work, and I'm under relocation so very little chance at home. I'll perform some install tests as soon as I'm able to, but it will not be for a few days.
Javier Palacios
Javier Palacios wrote:
Can you comment on the missing "$tree" in the seed file and how we can get the preseed and/or kernel options to reference the install source? This was from Friday's comment.
The value I get out from cobbler profile dumpvars --name=debian-xxx seems sensible, although the actual symlink on filesystem is wrong and points just to top ks_mirror. It might be because the code I'm using now is a bit evolved respect to current git contents.
Is there something you haven't sent to the ML yet? Your refactor patch is still in queue and I'll be looking at that next.
Anyway, and despite the fact that the kernel/initrd from cd might not be prepared for network install, there is plenty of work required for even partly automated installs. For example, the simple url --url=$tree statement in the kickstart sample is far away from debian's counterpart. FIrst of all, in the preseed the url is actually splitted in three different parts: the protocol, the host and the directory.
Seems to be doable with a trivial regex at worst :)
Right now I cannot perform any test at work, and I'm under relocation so very little chance at home. I'll perform some install tests as soon as I'm able to, but it will not be for a few days.
Javier Palacios _______________________________________________ cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
That's fine, no rush. Just wanted to see that it was in plan as it's going to be /veeeeeeerrrry/ nice to have :)
Thanks for all you have done so far!
(Enjoy moving, I know it is always fun to put everything in boxes and then take everything out of the same boxes.)
--Michael
Javier Palacios wrote:
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Actually recover the multi-import has been much easier than I expected. The patch attached works, at least for me.
I can import three distros as cobbler import --mirror=/media/distros/rhas/4.6 --name=rhas cobbler import --path=/media/distros/Fedora9/i386/ --name=f9_32 cobbler import --mirror=/media/distros/Fedora9/x86_64 --name=f9_64
or issue a single command cobbler import --mirror=/media/distros --name=allinone --breed=redhat
and all of them work.
Regarding the patch itself, a couple of comments. I've moved distro_adder method close to the distro add related method, but there is no change there. I have also modified item_distro.py, to remove breed and arch defaults. It was difficult to know why (and where) was the distro flavor set without os version until I realize that the redhat flavor was given on instantiation. It might be safer not to move that change into HEAD, but at least for devel branch I believe it is a good idea to have no such hardcoded defaults. The remaining changes are mostly intended as a first step to the real refactor, to make easier understand the changes than submitting the patch in the final form
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Thanks very much for the interest in this. A few small problems:
(A) This patch breaks "make test" on my machine as it tries to call "distro.set_arch()" with a value of None.
(B) Having a default for the item_distro.py values is a good thing, because if no argument is passed to "distro add" it can assume a reasonable default. At least for now redhat based distros constitute at least 95% of all cobbler usage. Yes, we do want this to change, but I don't believe we should require "--breed=redhat" on every distro add command. We could perhaps decide to be clever and choose the default based on the platform on which cobbler was running if we were concerned about that being a confusing feature on the other side of the fence. The changes to item_distro.py should not be applied, having arch defaults and breed defaults is there for usability reasons. If someone forgets to set them, they can set them later with an "edit" command, but this ensures there is always a valid value for these fields.
(C) Can you explain why the get_pxe_arch function, which was there to return the architecture of a specific point in the directory tree where the kernel+initrd is found, now returns a list? Are these architecture guesses? I can't see any ability for sharing of kernels between arches.
FYI: as for policy of what goes into HEAD, this is definitely something that we're looking at for devel, but we only roll bugfixes into HEAD. As this is new feature development devel will roll into head once 1.3.X is complete.
If you can take care of A, B, and C, I can look at applying this, as I think being able to process and import containing a mix of distribution trees on NFS is definitely useful. We just want to make extra sure everything that is currently there stays working.
I think I'm much more interested in seeing that we get fully automated answer file templates set up first, before we look at adding additional capabilities to the import code or look at refactoring. If we have the audience and the features, we can worry about code cleanup, but the main feature this needs (to have fully automated kickstarts straight out of the import) is still missing -- that is the main thing that "cobbler import" is intended to do.
--Michael
Michael DeHaan wrote:
Javier Palacios wrote:
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Actually recover the multi-import has been much easier than I expected. The patch attached works, at least for me.
I can import three distros as cobbler import --mirror=/media/distros/rhas/4.6 --name=rhas cobbler import --path=/media/distros/Fedora9/i386/ --name=f9_32 cobbler import --mirror=/media/distros/Fedora9/x86_64 --name=f9_64
or issue a single command cobbler import --mirror=/media/distros --name=allinone --breed=redhat
and all of them work.
Regarding the patch itself, a couple of comments. I've moved distro_adder method close to the distro add related method, but there is no change there. I have also modified item_distro.py, to remove breed and arch defaults. It was difficult to know why (and where) was the distro flavor set without os version until I realize that the redhat flavor was given on instantiation. It might be safer not to move that change into HEAD, but at least for devel branch I believe it is a good idea to have no such hardcoded defaults. The remaining changes are mostly intended as a first step to the real refactor, to make easier understand the changes than submitting the patch in the final form
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Thanks very much for the interest in this. A few small problems:
(A) This patch breaks "make test" on my machine as it tries to call "distro.set_arch()" with a value of None.
(B) Having a default for the item_distro.py values is a good thing, because if no argument is passed to "distro add" it can assume a reasonable default. At least for now redhat based distros constitute at least 95% of all cobbler usage. Yes, we do want this to change, but I don't believe we should require "--breed=redhat" on every distro add command. We could perhaps decide to be clever and choose the default based on the platform on which cobbler was running if we were concerned about that being a confusing feature on the other side of the fence. The changes to item_distro.py should not be applied, having arch defaults and breed defaults is there for usability reasons. If someone forgets to set them, they can set them later with an "edit" command, but this ensures there is always a valid value for these fields.
(C) Can you explain why the get_pxe_arch function, which was there to return the architecture of a specific point in the directory tree where the kernel+initrd is found, now returns a list? Are these architecture guesses? I can't see any ability for sharing of kernels between arches.
FYI: as for policy of what goes into HEAD, this is definitely something that we're looking at for devel, but we only roll bugfixes into HEAD. As this is new feature development devel will roll into head once 1.3.X is complete. If you can take care of A, B, and C, I can look at applying this, as I think being able to process and import containing a mix of distribution trees on NFS is definitely useful. We just want to make extra sure everything that is currently there stays working.
I think I'm much more interested in seeing that we get fully automated answer file templates set up first, before we look at adding additional capabilities to the import code or look at refactoring. If we have the audience and the features, we can worry about code cleanup, but the main feature this needs (to have fully automated kickstarts straight out of the import) is still missing -- that is the main thing that "cobbler import" is intended to do.
--Michael
Correction:
(A) was my fault, running "make eraseconfig" removed the objects that were causing that problem.
Other comments still apply.
Sorry for the confusion!
--Michael
(B) Having a default for the item_distro.py values is a good thing, because if no argument is passed to "distro add" it can assume a reasonable default. At least for now redhat based distros constitute these fields. [...] (C) Can you explain why the get_pxe_arch function, which was there to return the architecture of a specific point in the directory tree where the kernel+initrd is found, now returns a list? Are these architecture guesses? I can't see any ability for sharing of kernels between arches.
Here is the patch again, with redhat re-enabled as default, so actually fixing B. Regarding C, I can work that easily if you believe is better, although I might prefer not to drop the list yet, until a sensible guess could be done from the imported tree and to leave open the way for multiarch repositories and media.
Javier Palacios
Javier Palacios wrote:
(B) Having a default for the item_distro.py values is a good thing, because if no argument is passed to "distro add" it can assume a reasonable default. At least for now redhat based distros constitute these fields. [...] (C) Can you explain why the get_pxe_arch function, which was there to return the architecture of a specific point in the directory tree where the kernel+initrd is found, now returns a list? Are these architecture guesses? I can't see any ability for sharing of kernels between arches.
Here is the patch again, with redhat re-enabled as default, so actually fixing B. Regarding C, I can work that easily if you believe is better, although I might prefer not to drop the list yet, until a sensible guess could be done from the imported tree and to leave open the way for multiarch repositories and media.
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
The list stuff is fair (C), I didn't see that in the earlier patch. It seems to work fine.
I'll run tests shortly and let you know how things work out. Thanks!
--Michael
Javier Palacios wrote:
(B) Having a default for the item_distro.py values is a good thing, because if no argument is passed to "distro add" it can assume a reasonable default. At least for now redhat based distros constitute these fields. [...] (C) Can you explain why the get_pxe_arch function, which was there to return the architecture of a specific point in the directory tree where the kernel+initrd is found, now returns a list? Are these architecture guesses? I can't see any ability for sharing of kernels between arches.
Here is the patch again, with redhat re-enabled as default, so actually fixing B. Regarding C, I can work that easily if you believe is better, although I might prefer not to drop the list yet, until a sensible guess could be done from the imported tree and to leave open the way for multiarch repositories and media.
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Ok, just tested this...
Running on Fedora 9 (and using --available-as to save space, not because it was needed), I am unable to import a F-9 DVD.
[root@localhost cobbler]# mount -o loop /home/Fedora-9-i386-DVD.iso /tmp/loopy [root@localhost cobbler]# cobbler import --name=loopy --available-as=http://loopy/ --mirror=/tmp/loopy ---------------- (adding distros) - scanning /tmp/loopy/images/pxeboot for architecture info - architectures found : [] - scanning /tmp/loopy/images/xen for architecture info - architectures found : [] ---------------- (associating kickstarts) ---------------- (syncing)
[root@localhost cobbler]# cobbler profile list | grep loopy
Import of directory trees off of NFS, including multiple directory trees, seems to work.
[root@localhost cobbler]# cobbler import --name=IMP4 --mirror=/mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os/ --available-as=http://sparky ---------------- (adding distros) - creating new distro: IMP4-i386 - creating new profile: IMP4-i386 - creating new profile: rescue-IMP4-i386 - creating new distro: IMP4-xen-i386 - creating new profile: IMP4-xen-i386 ---------------- (associating kickstarts) /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os/, /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os, -1 - warning: possible symlink traversal?: /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os/, /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os, -1 - warning: possible symlink traversal?: /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os/, /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os, -1 - warning: possible symlink traversal?: /mnt/engarchive2/released/F-9/GOLD/Fedora/i386/os ---------------- (syncing)
If you can fix the Fedora ISO import case, I'll apply this.
It is looking like imports are "complicated" so if you do think we need seperate import code for Fedora/Red Hat/CentOS and Debian we can do that, and require a "--breed" to import.
That may eliminate the problems of breaking the RHEL use cases while working on the Debian ones and might make the Debian code simpler. If you want to continue down the current direction that is fine too, but if you can start testing with Fedora ISOs before submitting that would be helpful to me.
I was happy with our previous Debian import capability and still think the most important thing we can do to make it truly useful is to template out some good fully automated preseeds, and then look at this later if needed. I'm less interested in whether Debian can import in batch and so forth as I am as to whether it is fully automated out of the box and we ship some good sample preseeds to show people the way to go.
Thanks!
--Michael
Running on Fedora 9 (and using --available-as to save space, not because it was needed), I am unable to import a F-9 DVD. [...] Import of directory trees off of NFS, including multiple directory trees, seems to work.
I still need to have a test with the network_root thing, but I've found a bug in my patch that was probably the cause of failure and I didn't see before because I was focusing on multi-tree imports.
On function match_try_list there is an indentation problem in the final return statement. The line should have two less whitespaces, so the return is for the whole method and not for any of the loops.
The real problem with network_root is very likely on the pending FIXME on method get_new_tree_root, so I don't expect this to be a complex fix, but in any case I will repeat my tests with increased carefull before resubmitting
Javier Palacios
Running on Fedora 9 (and using --available-as to save space, not because it was needed), I am unable to import a F-9 DVD. [...] If you can fix the Fedora ISO import case, I'll apply this.
The first patch attached (#0) fixes the --network-root imports. After realize on the indentation bug it has been really straighforward.
I've increased my test cases with two network-root imports (a single fedora and a double fedora + RHAS tree). And as I have improved the test scripts, I'm completelly sure that the results of all the imports are not only the same that when patches are not applied, but even better as the os version is also discovered on multiple tree imports.
It is looking like imports are "complicated" so if you do think we need seperate import code for Fedora/Red Hat/CentOS and Debian we can do that, and require a "--breed" to import.
That was actually the idea, and the second patch (#1) is the first real step to that goal. It tries to encapsulate the distro add methods, and in brief will hold other switch-alike methods as scan_pkg_filename and set_variance.
Javier Palacios
P.S.: The patch are actually splited just to make easier to understand the code reorganization
I attach a new patch, to be applied after the previous ones, that encapsulates most of the kickstart_finder portions that depend on the breed.
I've been able to perform some debian install tests. Basically, the kernel/initrd from media are not able to perform a network install. They actually download the preseed file, but at a so late stage than is not very useful. So, kernel/initrd from network install specific media should be imported and used. Moreover, some checksum/pgp files that are required by the network install images are absent on media for local install, so importing will probably be useful as a first step towards a real mirror/repository.
Regarding the boot itself, the supplied sample.seed works, but some modifications are required. Instead of "ks", the kernel boot argument to point to the preseed file is named "url", and if we have multiple network interfaces, we must select it with "interface" instead of "ksdevice" (the word "auto" can be used instead of a named interface to leave the installer to select the interface). Up to here there is little surprise, and it's just about different keyword names. But for debian, some of the typical kickstart values must be supplied in advance. Specifically, we must use "locale" to set it and avoid the question to be asked. The only way I know to really preseed the keyboard type is to use "priority=critical" which just reduces the number of asked questions, and as a side effect also avoids to confirm the values for host and domain supplied by the DHCP server. These are the points that apply before the preseed file is downloaded.
To use a cobbler provided repository instead of a official debian mirror, some lines should be added to sample.seed : d-i mirror/country string enter information manually d-i mirror/http/hostname string cobbler.server d-i mirror/http/directory string /path/to/debian d-i mirror/http/proxy string apt-setup-udeb apt-setup/security_host string cobbler.server That will use http://cobbler.server/path/to/debian as source for packages, and the security updates from http://cobbler.server/debian-security (as fas a I know, only debian-security can be used as path for security updates).
Javier Palacios
Here is the "final" version of the patch to import debian media. The full one is against current git HEAD, while the incremental one should be applied after the three previous patches (0,1 and 3).
Only the repo finder part is not modified, because I want to have a look to allow creation and management of debian repositories.
As far as I see, only fixing proper ["tree"] is required for debian/ubuntu, but I'll not work on that until I'm able to arrange a second machine to perform real tests.
Javier Palacios
P.S. : Although they was not intended to, the patches for debian did solve issue 221 (cobbler import overwrites distro and profile configurations). Within add_entry, if condition 'existing_distro is not None', instead of printing "modifying existing distro" message, it raises an exception warning about that, so nothing is rewritten. Just returning might be goot, and a command line option to force overwriting might be required, but the exception at least disables unwanted removals.
Javier Palacios wrote:
Here is the "final" version of the patch to import debian media. The full one is against current git HEAD, while the incremental one should be applied after the three previous patches (0,1 and 3).
Only the repo finder part is not modified, because I want to have a look to allow creation and management of debian repositories.
As far as I see, only fixing proper ["tree"] is required for debian/ubuntu, but I'll not work on that until I'm able to arrange a second machine to perform real tests.
Javier Palacios
P.S. : Although they was not intended to, the patches for debian did solve issue 221 (cobbler import overwrites distro and profile configurations). Within add_entry, if condition 'existing_distro is not None', instead of printing "modifying existing distro" message, it raises an exception warning about that, so nothing is rewritten. Just returning might be goot, and a command line option to force overwriting might be required, but the exception at least disables unwanted removals.
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Thanks! I have a 1.2.5 release to get out today, then I'll take a look.
--Michael
Javier Palacios wrote:
Here is the "final" version of the patch to import debian media. The full one is against current git HEAD, while the incremental one should be applied after the three previous patches (0,1 and 3).
Only the repo finder part is not modified, because I want to have a look to allow creation and management of debian repositories.
As far as I see, only fixing proper ["tree"] is required for debian/ubuntu, but I'll not work on that until I'm able to arrange a second machine to perform real tests.
Javier Palacios
P.S. : Although they was not intended to, the patches for debian did solve issue 221 (cobbler import overwrites distro and profile configurations). Within add_entry, if condition 'existing_distro is not None', instead of printing "modifying existing distro" message, it raises an exception warning about that, so nothing is rewritten. Just returning might be goot, and a command line option to force overwriting might be required, but the exception at least disables unwanted removals.
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Thanks!!!
I'm testing this now/tomorrow so I should have some info on this soon.
--Michael
Javier Palacios wrote:
Here is the "final" version of the patch to import debian media. The full one is against current git HEAD, while the incremental one should be applied after the three previous patches (0,1 and 3).
Only the repo finder part is not modified, because I want to have a look to allow creation and management of debian repositories.
As far as I see, only fixing proper ["tree"] is required for debian/ubuntu, but I'll not work on that until I'm able to arrange a second machine to perform real tests.
Javier Palacios
P.S. : Although they was not intended to, the patches for debian did solve issue 221 (cobbler import overwrites distro and profile configurations). Within add_entry, if condition 'existing_distro is not None', instead of printing "modifying existing distro" message, it raises an exception warning about that, so nothing is rewritten. Just returning might be goot, and a command line option to force overwriting might be required, but the exception at least disables unwanted removals.
Ok, looked this over, applied it (using the "full" patch), and then hacked on it a bit:
Here's some of the changes I made to this, not all related to your changes.
I did not add any functions so these fall under the area of bugfixes to weird corner cases and adding documentation. There is probably more to do so if others would like to test that would be useful.
Most of these tests were of the form "import a large amount of trees on NFS under a common root, whether RHEL 3 or RHEL 4". These were RHEL trees but should largely cover the CentOS use cases as well.
-- I had to add another line to the signature checker to make it see RHEL 4 trees on NFS again. -- Added recognition for RHEL 3 Desktop trees (3Desktop) in set_variance code -- I had to add "kernel-smp" to the list of kernel RPMs to look for as it wasn't adding x86_64 architectures as is -- Changed a lot of "raise" conditions to just warnings so they didn't stop a large import that contained some trees it didn't yet deal with (i.e. ppc, s390 (not s390x)). -- Removed some "raise" lines with code immediately below it that would never be executed. -- If a name is used, warn, don't stop, but don't edit it either (similar to above, which you mentioned) -- Some extra code formatting (whitespace, etc), added a lot of comments to methods I originally had uncommented (long time coming from me) -- converted some module comments from # to docstrings
I haven't gotten around to testing the Debian/Ubuntu layers yet, the above was all related to RHEL/Fedora testing.
General comments
-- output seems rather "loud" as compared to previously, which was already loud, I need to make this run quieter, perhaps with a --verbose mode when debug is required -- especially over NFS, four "os.path.walk" calls leads to a lot of os.stat calls underneath -- supporting treeinfo and discinfo should optomize this further (this was an existing problem too)
I agree, I think the next thing is definitely setting up the "tree" variable and showing that we have fully automatic installs straight from an import and then picking the new distributions from the PXE menus. Sounds good!
I realize you don't have the huge library of NFS trees to test against though hopefully the hard parts are over and all we need to do now is auto-assign a working tree and have a fully automated template or two to go with it. That would be pretty slick. (These are also not the most common import use cases, but are the ones that beat on it the hardest).
This is definitely one of the more gnarly areas of code (seeing it has to be kind of psychic to find things out), so I greatly appreciate your willingness to work on this. It is also good to see the distro specific bits abstracted out into different classes.
Here's my changes on top, a few more are likely coming as I continue testing this.
http://git.fedorahosted.org/git/cobbler?p=cobbler;a=commitdiff;h=b224f84192a...
Thanks!
--Michael
Michael DeHaan wrote:
Javier Palacios wrote:
Here is the "final" version of the patch to import debian media. The full one is against current git HEAD, while the incremental one should be applied after the three previous patches (0,1 and 3).
Only the repo finder part is not modified, because I want to have a look to allow creation and management of debian repositories.
As far as I see, only fixing proper ["tree"] is required for debian/ubuntu, but I'll not work on that until I'm able to arrange a second machine to perform real tests.
Javier Palacios
P.S. : Although they was not intended to, the patches for debian did solve issue 221 (cobbler import overwrites distro and profile configurations). Within add_entry, if condition 'existing_distro is not None', instead of printing "modifying existing distro" message, it raises an exception warning about that, so nothing is rewritten. Just returning might be goot, and a command line option to force overwriting might be required, but the exception at least disables unwanted removals.
Ok, looked this over, applied it (using the "full" patch), and then hacked on it a bit:
Here's some of the changes I made to this, not all related to your changes.
I did not add any functions so these fall under the area of bugfixes to weird corner cases and adding documentation. There is probably more to do so if others would like to test that would be useful.
Most of these tests were of the form "import a large amount of trees on NFS under a common root, whether RHEL 3 or RHEL 4". These were RHEL trees but should largely cover the CentOS use cases as well.
-- I had to add another line to the signature checker to make it see RHEL 4 trees on NFS again. -- Added recognition for RHEL 3 Desktop trees (3Desktop) in set_variance code -- I had to add "kernel-smp" to the list of kernel RPMs to look for as it wasn't adding x86_64 architectures as is -- Changed a lot of "raise" conditions to just warnings so they didn't stop a large import that contained some trees it didn't yet deal with (i.e. ppc, s390 (not s390x)). -- Removed some "raise" lines with code immediately below it that would never be executed. -- If a name is used, warn, don't stop, but don't edit it either (similar to above, which you mentioned) -- Some extra code formatting (whitespace, etc), added a lot of comments to methods I originally had uncommented (long time coming from me) -- converted some module comments from # to docstrings
I haven't gotten around to testing the Debian/Ubuntu layers yet, the above was all related to RHEL/Fedora testing.
General comments
-- output seems rather "loud" as compared to previously, which was already loud, I need to make this run quieter, perhaps with a --verbose mode when debug is required -- especially over NFS, four "os.path.walk" calls leads to a lot of os.stat calls underneath -- supporting treeinfo and discinfo should optomize this further (this was an existing problem too)
I agree, I think the next thing is definitely setting up the "tree" variable and showing that we have fully automatic installs straight from an import and then picking the new distributions from the PXE menus. Sounds good!
I realize you don't have the huge library of NFS trees to test against though hopefully the hard parts are over and all we need to do now is auto-assign a working tree and have a fully automated template or two to go with it. That would be pretty slick. (These are also not the most common import use cases, but are the ones that beat on it the hardest).
This is definitely one of the more gnarly areas of code (seeing it has to be kind of psychic to find things out), so I greatly appreciate your willingness to work on this. It is also good to see the distro specific bits abstracted out into different classes.
Here's my changes on top, a few more are likely coming as I continue testing this.
http://git.fedorahosted.org/git/cobbler?p=cobbler;a=commitdiff;h=b224f84192a...
Thanks!
--Michael
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
Another thought is that if we can break things down into just one os.path.walk (or at least make sure we're not walking the same files twice), that would be a very useful speedup.
--Michael
On Wed, Oct 1, 2008 at 12:30 AM, Michael DeHaan mdehaan@redhat.com wrote:
Michael DeHaan wrote:
Another thought is that if we can break things down into just one os.path.walk (or at least make sure we're not walking the same files twice), that would be a very useful speedup.
I've had a look at this, and arch_walker and repo_scanner could be very probably avoided. Both are used to find files that we exactly know where they are within the imported tree, so a directory listing will suffice in the worst case. I'll work that when I come back to import after finishing the debian repo thing.
Javier Palacios
(B) Having a default for the item_distro.py values is a good thing, because if no argument is passed to "distro add" it can assume a reasonable default. At least for now redhat based distros constitute at least 95% of all cobbler usage. Yes, we do want this to change, but
The pre'sence or values for distro defaults have no impact in the import code. Just don't apply the patch to item_distro.py would suffice. I'll keep the empty defaults on my devel copy, and clean them from future patch submissions.
(C) Can you explain why the get_pxe_arch function, which was there to return the architecture of a specific point in the directory tree where the kernel+initrd is found, now returns a list? Are these architecture guesses? I can't see any ability for sharing of kernels between arches.
Basically, they return an array to list every arch found, when learn_arch_from_tree is invoked. At least for debian, multiple arches can live in the same media or repository, and returning the first one might cause inconsistency. Later on I realized that it is a method to guess the archs supported at the imported media, and that the list can be checked against command line args (for example) to also increase consistency, although that's only on the way. Although the behaviour is exactly the same than the old one when only an architecture is found, the old behaviour is easy to recover, raising an exception if multiple arches found and return just the first item, so turning unnecesary the for loop on the get_pxe_arch call,
Javier Palacios
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Excellent.
I'm actually downloading Solaris 10 x86 now, to see how hard it would be to add in jumpstart support to cobbler (I'm thinking, not very hard based on the addition of the debian/ubuntu stuff).
James Cammarata wrote:
I've been revisiting the previous import code, and I'm working in a small refactoring that will probably re-enable importing of multiple distros at the same time, at the price of all of them sharing a single breed specified at command line, and will also make easier to add other distros.
Excellent.
I'm actually downloading Solaris 10 x86 now, to see how hard it would be to add in jumpstart support to cobbler (I'm thinking, not very hard based on the addition of the debian/ubuntu stuff).
Excellent...
Yes, I think it would be very easy to do, possibly would require some small changes to things like pxegen.py and the DHCP management modules, plus adding a new breed into item_distro.py.
--Michael
On Thu, Sep 4, 2008 at 10:53 PM, Michael DeHaan mdehaan@redhat.com wrote:
(B) When attempting to correct this (removing the archmap code), the following command now fails to work:
make devinstall && make eraseconfig && cobbler import --name=RHEL3U3 --mirror=/mnt/engarchive2/released/RHEL-3/U3/AS/ --available-as=http://foo [...] And imports all of the distributions, creating objects for each. For instance, it will find the appropriate tree distros: [...]
Now I see clearly the problem. I thought that the idea was to import a single distribution from a single media, but if understand right, you have multiple distributions stored under a single directory, and while importing with that root directory as "mirror", you actually get a separate import for every single distribution found there. If that is the case, and I'm pretty sure it is, then I can tell you exactly why the patch failed, and why the new one will also fail.
What import is currently doing is to find every vmlinuz/initrd pair. Then, checks the path name against the known architectures, and if there is no match (there isn't for unsupported archs), tries to find it from the included packages.
Here is where the problem comes, in method learn_arch_from_tree(dirname). In the initial form it starts searching two directories above where the kernel and initrd where found. That uses two facts : - The vmlinuz & initrd at isolinux were explicitly rejected - Current media for RedHat based distros hosts all the remaining vmlinux and initrd at images/xxx/ That explains why the path walking starts two nodes above where the distro was found, and is ok for RedHat, but that does not hold for debian media. On debian media that is not true, and they are just one directory below the media root. So, if you jump two directories you reach the ks_mirror directory, which is not the right place to start. To avoid this, I used the code from get_proposed_name to find the root of the imported tree, and search from there (although I now realize that ks_mirror/name might have tree from multiple media).
There is other point related (on arch_walker method) where the import could fail, even for a real single distribution media. Instead of returning only one architecture, the patched versions returns the list with all the architectures found, and raises an exception if there are more than one. That is intended to reject importing of media for multiple architectures, that exists for debian and might have greater impact on other portions of cobbler not related to import.
The new archwalker in this patch seems to not identify an arch for "ls /mnt/engarchive2/released/RHEL-3/U3/" which means it stopped too early, which is why I am suggesting that Debian support be added without trying to refactor/cleanup the code -- basically we just need to keep it simple. Try to get the bare minimum to add the import in there, so
I'm not sure if the points I explained above are clear, but I really don't think that extensions to other distros could be done without some refactoring. The same pieced of code will hardly allow import multiple RedHat trees at the same time, with the ability to import a variety of distros. Moreover, the presence of some command line arguments (arch and breed) does not fit well if many of them could be imported with a single run. The only "easy" path that I see is disable the multi-tree import (hopefully, only temporarily), or at least add a new command line option to enable it (thought it's unclear how to handle this in the code).
Javier Palacios
Javier Palacios wrote:
Hi,
I've started to work in order to allow debian provisioning with cobbler. I send attached two patches.
The first on is actually not strictly related to debian, and it's a reorganization of the archs management, to simplify it and ease smooth addition of other distros: debian calls amd64 to x86_64, and the idea is to keep the RedHat names as canonical ones.
The second one does actually allow to import the content of a debian CD/DVD and create the corresponding distro. It also creates the profile breeded as debian, although it might not be usable yet.
There are a couple of open questions that might affect other parts of cobbler or where just ideas are required.
- the kickstarts for debian. They are called "preseed", and are by far
much more complex than a standard kickstart. Although not strictly required, it might be better to have new names for both ks files and directories,
Cobbler used to have the alias "--answer-file" for "--kickstart" though if added it should once again only be an alias, as opposed to having different field storage for the answer file. Essentially this is what we do already for --breed=debian (preseed) or --breed=suse (AutoYAST), both being supported already. For Ubuntu kickseeds, you just use --breed=redhat and need to be very careful with the contents of the kickseed.
I have also been told kickseeds have been applied to upstream Debian, but haven't been able to validate this or know whether they are in fact a good option. I'll leave that for other people to answer.
The other thing this import should do, that is very important, is to package a good default preseed so users can get a fully automated setup out of the box. Just creating the distro is pretty easy, the magic is in setting up something for a user who doesn't know how to write one that installs a base package set.
but I think that they can be easyly added to PXE boot options. To make things worst, ubuntu does allow both the standard debian preseed and kickstart files.
- the package repositories. Debian follows a quite different approach,
and a single tree can hold many versions and different architectures. Allowing a cobbler distro to have multiple architectures might reduce the disk usage and simplify the repository updating. Due to this fact, a single debian media can install multiple archs, complexing a little bit to import the media.
I've heard adding support for "cobbler repo" style objects would be difficult here. One option is to just not support them and require that they be managed out of band.
These are the only two points I see on the close horizon, but there are probably more. It anyone has concerns or ideas, they are welcome. And if anyone can actually test the deployment (when it matures a little bit), much better.
Javier Palacios
cobbler mailing list cobbler@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/cobbler
cobbler@lists.fedorahosted.org