Hi all,
One of the tasks I've been working on is #3178, "Assess the viability of importing and launching images from a openstack provider." We don't have many OpenStack implementation tasks in this sprint, but I was kind of roaring to see something working, so I wanted to sort of take inventory of where things stood today.
With the patches adding support applied to master a bit ago, I pulled up a Rails console and saw how far I could get. [1] lists the steps I took.
The first obstacle I hit was some exception in the Deltacloud logs, raised when I tried to list instances[2]. Marios helped me debug this and indicated that the problem was that I was using an older version of the Deltacloud package; it was using the Rackspace driver versus the OpenStack one. (See the filename in line 20 of the link.)
Until this is updated in the next Deltacloud release, I worked around this by building a local gem for Deltacloud and restarting with that. That much worked great.
My next issue was that we do not appear to have anything in place to collect any credentials for those setting up an OpenStack provider account. Thus I cheated and used the Rails console for now, talking to Deltacloud directly. And with that, basic Deltacloud API operations work -- I can list images and instances, for example.
Complicating things, the credentials we need are semi-variable. If you are running Keystone, we need to collect your username, password, and tenant name. If you're not running Keystone, we don't need your tenant name. (OpenStack also allows authentication via key + secret key, but Deltacloud does not presently support this.) I'm not sure what the most elegant solution is here just yet, nor if there's a good way to proceed without provider-specific logic. As far as Deltacloud is concerned, the tenant name is just added to the username, so we could do the same, except it might not be particularly clear to end users.
I attempted to add Credential Definitions and add a provider account, but that resulted in me receiving some exceptions about hardware profiles.
I'd like to investigate further, but I think I'm fairly far afield from my stated objective of investigating the viability of importing images. It may be that I'm just around the bend from getting it working, but I'm starting to think that the conclusion here is that, in its present form, the answer to the task is "Not very viable at the moment." I'd be happy to carry on, but only if people think it's a worthwhile endeavor before Factory adds support for building for OpenStack.
-- Matt
[1] https://gist.github.com/2431166 [2] https://gist.github.com/2431171
On 04/23/2012 11:11 PM, Matt Wagner wrote:
I'd like to investigate further, but I think I'm fairly far afield from my stated objective of investigating the viability of importing images. It may be that I'm just around the bend from getting it working, but I'm starting to think that the conclusion here is that, in its present form, the answer to the task is "Not very viable at the moment." I'd be happy to carry on, but only if people think it's a worthwhile endeavor before Factory adds support for building for OpenStack.
IMO it seems worthwhile to continue moving towards adding support for OpenStack in whatever manner we can as that'd be a great feature to use to demonstrate Aeolus' capabilities.
Going forward we are working to make the image management bits conductor pluggable so we can swap things in and out. AFAIK it looks like boxgrinder supports building images for openstack so we should be able to use that for that scenario until imagefactory supports openstack (at which point we can just enable the flag to swap that back in)
-Mo
On 24/04/12 11:44 +0200, Mo Morsi wrote:
On 04/23/2012 11:11 PM, Matt Wagner wrote:
I'd like to investigate further, but I think I'm fairly far afield from my stated objective of investigating the viability of importing images. It may be that I'm just around the bend from getting it working, but I'm starting to think that the conclusion here is that, in its present form, the answer to the task is "Not very viable at the moment." I'd be happy to carry on, but only if people think it's a worthwhile endeavor before Factory adds support for building for OpenStack.
IMO it seems worthwhile to continue moving towards adding support for OpenStack in whatever manner we can as that'd be a great feature to use to demonstrate Aeolus' capabilities.
Agree that we should keep pushing on this, and it seems to me solidifying deltacloud support (sounds like some bugs there) and adding proper credentials for the provider type should be the first targets. It may be that the latter issue could be solved by querying the dc api for a given openstack provider to find out what the supported/expected credentials are for that specific provider, since they may vary between openstack versions. If this does not currently exist in deltacloud, I think we have a feature request, as that is within deltacloud's purvue of abstracting away the differences and/or making needed differences discoverable (at least it seems so, imo).
I may be remembering wrong, but I could have sworn marios told me that the key/secret auth was actually the older one, and has been supported for some time, with keystone being the newer less tested addition. CC'ing him for the correct answer.
Going forward we are working to make the image management bits conductor pluggable so we can swap things in and out. AFAIK it looks like boxgrinder supports building images for openstack so we should be able to use that for that scenario until imagefactory supports openstack (at which point we can just enable the flag to swap that back in)
-Mo
This point I have to disagree with. While we _are_ in the process of making our factory interaction into a rails engine, that does not imply any plans to switch _away_ from factory, even temporarily. In factory v2, it will actually also serve some of the functionality that iwhd currently provides us, so swapping it out entirely would be detrimental to the whole stack. In fact, factory _mostly_ has openstack support already, from what I understand. Iirc, there has been a working rackspace impl for quite some time, even though we have not consumed it from conductor. CC'ing Ian and Steve so they can clarify where that stands, and if they have any time/plans to look at it, or need some help from the community if that is a desired feature. From what I recall there were an unknown number of 'tweaks' needed to deal with generic openstack vs rackspace specifically (perhaps unknown only to me).
-j
On 25/04/2012, at 2:56 AM, Jason Guiditta wrote: <snip>
Iirc, there has been a working rackspace impl for quite some time, even though we have not consumed it from conductor. CC'ing Ian and Steve so they can clarify where that stands, and if they have any time/plans to look at it, or need some help from the community if that is a desired feature. From what I recall there were an unknown number of 'tweaks' needed to deal with generic openstack vs rackspace specifically (perhaps unknown only to me).
If it's helpful, the Dell OpenStack Community Manager guy said Hi to us a while ago, and seems pretty friendly.
Would it be useful to ask him about getting some Dev accounts or something onto their cloud, so we can look to adding support?
Regards and best wishes,
Justin Clift
-- Aeolus Community Manager http://www.aeolusproject.org
Going forward we are working to make the image management bits conductor pluggable so we can swap things in and out. AFAIK it looks like boxgrinder supports building images for openstack so we should be able to use that for that scenario until imagefactory supports openstack (at which point we can just enable the flag to swap that back in)
-Mo
This point I have to disagree with. While we _are_ in the process of making our factory interaction into a rails engine, that does not imply any plans to switch _away_ from factory, even temporarily. In factory v2, it will actually also serve some of the functionality that iwhd currently provides us, so swapping it out entirely would be detrimental to the whole stack. In fact, factory _mostly_ has openstack support already, from what I understand. Iirc, there has been a working rackspace impl for quite some time, even though we have not consumed it from conductor.
We do indeed have some rarely-used and sparsely tested support for snapshot style RackSpace builds.
However, what I gather will be more interesting to people is the ability to upload scratch-built images via Glance. I'm working on support for that now and it should be completed in our current sprint, which ends in 2.5 weeks. (It may be available in git much sooner than that.)
A larger question is what specific OpenStack configurations we wish to support. It appears that OpenStack can use just about every commonly deployed hypervisor:
http://docs.openstack.org/cactus/openstack-compute/admin/content/selecting-a...
I'm pretty sure we won't be able to produce a single binary image that can run on all of those. For example, we know that a RHEL5 image must be configured to boot a different kernel in the Xen environment. It's also unclear to what extent OpenStack is able to munge the image format in a multiple hypervisor configuration. Can it, for example, take a qcow2 image and transform it to VMDK in order to launch an ESX VM?
CC'ing Ian and Steve so they can clarify where that stands, and if they have any time/plans to look at it, or need some help from the community if that is a desired feature. From what I recall there were an unknown number of 'tweaks' needed to deal with generic openstack vs rackspace specifically (perhaps unknown only to me).
-j
On 04/26/2012 03:27 PM, Ian McLeod wrote:
Going forward we are working to make the image management bits conductor pluggable so we can swap things in and out. AFAIK it looks like boxgrinder supports building images for openstack so we should be able to use that for that scenario until imagefactory supports openstack (at which point we can just enable the flag to swap that back in)
-Mo
This point I have to disagree with. While we _are_ in the process of making our factory interaction into a rails engine, that does not imply any plans to switch _away_ from factory, even temporarily. In factory v2, it will actually also serve some of the functionality that iwhd currently provides us, so swapping it out entirely would be detrimental to the whole stack. In fact, factory _mostly_ has openstack support already, from what I understand. Iirc, there has been a working rackspace impl for quite some time, even though we have not consumed it from conductor.
What is the feasibility of using multiple image management backends in conductor at the same time? Or at least allowing the admin to select which ones are deployed?
It might make sense to make use of these on a per-provider-type basis, as I can foresee a situation in which certain image generation utilities have limited or no support for particular cloud providers (for example those that provide proprietary/restricted interfaces). It would be nice to allow the conductor end-user to use those on an on-demand basis, making use of the native image generation / management tooling or other in those situations.
We do indeed have some rarely-used and sparsely tested support for snapshot style RackSpace builds.
However, what I gather will be more interesting to people is the ability to upload scratch-built images via Glance. I'm working on support for that now and it should be completed in our current sprint, which ends in 2.5 weeks. (It may be available in git much sooner than that.)
Cool, would a release of imagefactory in Fedora be able to be cut after that? Would be the easiest way for us to leverage those new capabilities from Conductor.
A larger question is what specific OpenStack configurations we wish to support. It appears that OpenStack can use just about every commonly deployed hypervisor:
http://docs.openstack.org/cactus/openstack-compute/admin/content/selecting-a...
I'm pretty sure we won't be able to produce a single binary image that can run on all of those. For example, we know that a RHEL5 image must be configured to boot a different kernel in the Xen environment. It's also unclear to what extent OpenStack is able to munge the image format in a multiple hypervisor configuration. Can it, for example, take a qcow2 image and transform it to VMDK in order to launch an ESX VM?
If we added support to run the latest Fedora / RHEL on KVM (for example) would we be able to make use of the native tooling to build images for other non-supported scenarios (in the manner described above). This way we'd be able to leverage imagefactory for the scenarios it'd make sense to support while allowing the end user to use alternative solutions on a 'you're-on-your-own' basis.
-Mo
----- Original Message -----
Hi all,
One of the tasks I've been working on is #3178, "Assess the viability of importing and launching images from a openstack provider." We don't have many OpenStack implementation tasks in this sprint, but I was kind of roaring to see something working, so I wanted to sort of take inventory of where things stood today.
With the patches adding support applied to master a bit ago, I pulled up a Rails console and saw how far I could get. [1] lists the steps I took.
The first obstacle I hit was some exception in the Deltacloud logs, raised when I tried to list instances[2]. Marios helped me debug this and indicated that the problem was that I was using an older version of the Deltacloud package; it was using the Rackspace driver versus the OpenStack one. (See the filename in line 20 of the link.)
Until this is updated in the next Deltacloud release, I worked around this by building a local gem for Deltacloud and restarting with that. That much worked great.
Thanks for pushing on this! There also exists a story in the current sprint to get a community release underway so it is probably a good time to gather dep updates so we can make a coordinated push to a new set of fedora rpms.
My next issue was that we do not appear to have anything in place to collect any credentials for those setting up an OpenStack provider account. Thus I cheated and used the Rails console for now, talking to Deltacloud directly. And with that, basic Deltacloud API operations work -- I can list images and instances, for example.
Complicating things, the credentials we need are semi-variable. If you are running Keystone, we need to collect your username, password, and tenant name. If you're not running Keystone, we don't need your tenant name. (OpenStack also allows authentication via key + secret key, but Deltacloud does not presently support this.) I'm not sure what the most elegant solution is here just yet, nor if there's a good way to proceed without provider-specific logic. As far as Deltacloud is concerned, the tenant name is just added to the username, so we could do the same, except it might not be particularly clear to end users.
I attempted to add Credential Definitions and add a provider account, but that resulted in me receiving some exceptions about hardware profiles.
I'd like to investigate further, but I think I'm fairly far afield from my stated objective of investigating the viability of importing images. It may be that I'm just around the bend from getting it working, but I'm starting to think that the conclusion here is that, in its present form, the answer to the task is "Not very viable at the moment." I'd be happy to carry on, but only if people think it's a worthwhile endeavor before Factory adds support for building for OpenStack.
-- Matt
[1] https://gist.github.com/2431166 [2] https://gist.github.com/2431171
aeolus-devel@lists.fedorahosted.org