On Tue, 2011-05-24 at 11:35 -0400, Scott Seago wrote:
On 05/24/2011 10:52 AM, Mark McLoughlin wrote:
> Hey,
>
> Apologies for another long email, but here's my proposal for how to
> proceed with the image building CLI etc.
>
> Cheers,
> Mark.
>
> = Terminology =
>
> - Provider - a cloud instance, e.g. a RHEV-M installation or an EC2
> region.
>
> - Target/Provider Type - the cloud type, irrespective of the specific
> instance e.g. RHEV-M vs EC2.
>
> - Provider Account - a set of user credentials for a specific
> provider.
>
> - Provider Image - these are images which have been pushed to, or
> imported from, a provider and are available for use by instances.
>
> - Target Image - these are images which have been built for a
> specific target and are available to be pushed to multiple
> providers.
>
> - Image Build - these are objects which track all target and
> provider images built from a template at a certain time, all of
> which should be equivalent to each other.
>
OK this is the only change from what I had in my last writeup. Image
Build takes the place of Image Version -- this makes sense -- I'll just
need to update the redmine task accordingly.
Yep, sorry - I'm having a terminology death match and "build" is a shade
ahead of "version" :)
> - Image - these are metadata objects which describe the
purpose of
> the image, the parameters it takes, the set of available builds of
> the image and the latest build.
>
> = Overview =
>
> As described previously[1], the short/medium term plan is that Aeolus
> will:
>
> - Have well documented deployable and template description formats
> with plenty of examples. Deployable authors will create these
> descriptions manually or with the help of simple command line
> tools and store them as they fit (e.g. in their own git repo)
>
> - Support launching deployables in conductor simply by allowing the
> user to supply a URL to a deployable description, or choosing a
> URL from a list populated by an admin.
>
> - Include a set of command line tools for building images from a
> template description. This tools will also allow deployable
> authors to list available images available to be referenced in
> their deployables.
>
> - Store images and their related metadata in IWHD. Conductor will
> use IWHD queries to resolve deployables' image references and
> allow users to launch individual images directly.
>
> - Use image factory to build and import images, including the
> updating of IWHD metadata about the images.
>
> = IHWD Metadata =
>
> The following IWHD buckets will be used, each one for a different
> object type:
>
> images:
> - object_type == "image"
> - object_body: XML document with image description and parameters,
> for deployable authors and simple instance launch from UI
> - uuid: UUID
> - latest_build: build UUID
>
> builds:
> - object_type == "build"
> - object_body: empty
> - uuid: UUID
> - image: UUID of the image object
> - parent: UUID of previous build
>
Is the parent set automatically? i.e. for an initial build (image
doesn't exist before build), latest_build is this new build, and build
parent is null. For any subsequent build for an image, we always update
latest_build to be this new build, and the new build's parent is set to
what was previously the latest build.
Exactly
> target_images:
> - object_type == "target_image"
> - object_body: target specific image, for upload builds; empty for
> snapshot builds
> - uuid: UUID
> - build: UUID of the build object
> - icicle: UUID of icicle object
> - template: UUID of template object
> - target: ec2, rhev-m, etc.
> - target_parameters: target specific data stashed here for the push
> stage
>
> provider_images:
> - object_type == "provider_image"
> - object_body: empty
> - uuid: UUID
> - target_image: UUID of image
> - provider: e.g. ec2-us-east-1, rhev-m site
> - icicle == "none"
> - target_identifier: target specific ID for the image
>
Hmm. We should clarify with the image factory team, but in your model
here, icicle is associated with target images but not provider images.
In the current conductor model, we have icicle on provider images but
not target images. So we should make sure we have it right (or fix
accordingly)
I've taken this directly from image factory code e.g.
icicle_id = self.warehouse.store_icicle(self.output_descriptor)
metadata = dict(template=template_id, target=self.target, icicle=icicle_id,
target_parameters=target_parameters)
self.warehouse.store_target_image(self.image_id, self.image, metadata=metadata)
...
metadata = dict(target_image=image_id, provider=provider, icicle="none",
target_identifier=ami_id)
self.warehouse.create_provider_image(self.image_id, metadata=metadata)
> templates:
> - object_type == "template"
> - object_body:<template/> document supplied for image build
> - uuid: UUID
>
> icicles:
> - object_type == "icicle"
> - object_body:<icicle/> document
> - uuid: UUID
>
> This fairly closely reflects the metadata currently stored by image
> factory in IWHD, with the following changes:
>
> - the current "image" type is renamed to "target_type", along
with
> the bucket and image reference on provider images[2]
>
> - new image and build object types and buckets are introduced
>
> - a build reference is added to the target image type
>
> = Image Factory APIs =
>
> Image factory currently supports building target images and pushing
> provider images, along with the appropriate updating of image
> warehouse metadata.
>
> In order to allow the CLI to fire-and-forget, we will need to add new
> build and push APIs which can handle multiple images at once.
>
> These would look like:
>
> - image(image_id, template, targets[])
> + builds an image for the supplied targets
> + image_id should be ommitted, if a new image is required
> + template and targets may be ommitted if a previous build
> exists and the previous template and targets will be used
>
> - push(image_id, providers, credentials)
> + push the image the supplied providers
> + assumes factory can figure out which target image is
> appropriate for each provider
> + credentials argument is a<provider_credentials/> document
>
We need to work out how we intend to support the case where conductor
has more than one account on a given provider (i.e. in ec2-us-east we're
using 2 different provider accounts w/ different credentials).
Do we scope ProviderImages to a single provider account and upload for
each, or do we scope them to the provider -- in which case we upload
only once but then we have to worry about sharing the image with all
provider accounts we're using in conductor.
I have no idea :(
If we want to scope ProviderImages to provider accounts, we'd include a
provider account identifier in the provider image object
> Also, we'll need to rename image factory's current
concept of an image
> as a target image. Some initial work on that here[3].
>
> = Conductor APIs =
>
> Conductor needs APIs to support the following:
>
> 1) List the available provider types in an XML doc
> 2) List the available providers in an XML doc
> 3) Dump a<provider_credentials/> document encapsulating all of the
> available provider accounts
I'd like a little more clarification on these. Where are we using them,
and what parameters should these take (and, any more specifics needed on
the return xml doc).
Okay, these are all used by the image builder CLI
(1) and (2) for the "targets" and "providers" commands. You're
just
listing the available provider types and providers. No parameters
needed.
(3) is making ProviderAccount.build_credentials() available through the
API. We need to pass these credentials to image factory's push method.
> The latter API would be subject to authentication and access
control.
>
Actually, all of the conductor APIs should be subject to this -- i.e.
even listing providers should be filtered based on what providers the
calling user is allowed to see.
Even provider types?
Providers makes some sense, I guess ... I originally wrote "the latter
two APIs" :-)
I'd just prefer to avoid having the CLI user authenticate where it isn't
really necessary.
Thanks,
Mark.