On Tue, 2011-05-24 at 15:52 +0100, Mark McLoughlin wrote:
Hey,
Apologies for another long email, but here's my proposal for how to
proceed with the image building CLI etc.
Cheers,
Mark.
= Terminology =
- Provider - a cloud instance, e.g. a RHEV-M installation or an EC2
region.
- Target/Provider Type - the cloud type, irrespective of the specific
instance e.g. RHEV-M vs EC2.
- Provider Account - a set of user credentials for a specific
provider.
- Provider Image - these are images which have been pushed to, or
imported from, a provider and are available for use by instances.
- Target Image - these are images which have been built for a
specific target and are available to be pushed to multiple
providers.
- Image Build - these are objects which track all target and
provider images built from a template at a certain time, all of
which should be equivalent to each other.
- Image - these are metadata objects which describe the purpose of
the image, the parameters it takes, the set of available builds of
the image and the latest build.
= Overview =
As described previously[1], the short/medium term plan is that Aeolus
will:
- Have well documented deployable and template description formats
with plenty of examples. Deployable authors will create these
descriptions manually or with the help of simple command line
tools and store them as they fit (e.g. in their own git repo)
- Support launching deployables in conductor simply by allowing the
user to supply a URL to a deployable description, or choosing a
URL from a list populated by an admin.
- Include a set of command line tools for building images from a
template description. This tools will also allow deployable
authors to list available images available to be referenced in
their deployables.
- Store images and their related metadata in IWHD. Conductor will
use IWHD queries to resolve deployables' image references and
allow users to launch individual images directly.
- Use image factory to build and import images, including the
updating of IWHD metadata about the images.
= IHWD Metadata =
The following IWHD buckets will be used, each one for a different
object type:
images:
- object_type == "image"
- object_body: XML document with image description and parameters,
for deployable authors and simple instance launch from UI
- uuid: UUID
- latest_build: build UUID
builds:
- object_type == "build"
- object_body: empty
- uuid: UUID
- image: UUID of the image object
- parent: UUID of previous build
target_images:
- object_type == "target_image"
- object_body: target specific image, for upload builds; empty for
snapshot builds
- uuid: UUID
- build: UUID of the build object
- icicle: UUID of icicle object
- template: UUID of template object
I don't get why this ^ doesnt go with Image object?
- target: ec2, rhev-m, etc.
We need to be careful of magic string here, I propose these values come
from deltacloud api
- target_parameters: target specific data stashed here for the
push
stage
provider_images:
- object_type == "provider_image"
- object_body: empty
- uuid: UUID
- target_image: UUID of image
Do you actually mean uuid of image here, or of target image? As imcleod
said in his reply, this probably need to be where the icicle ref is
required, and target image is an optional one (to have a ref to icicle).
- provider: e.g. ec2-us-east-1, rhev-m site
- icicle == "none"
Is this ^ really just a literal string? If so, why have it? If not,
does this just mean it is an optional association?
- target_identifier: target specific ID for the image
templates:
- object_type == "template"
- object_body: <template/> document supplied for image build
- uuid: UUID
icicles:
- object_type == "icicle"
- object_body: <icicle/> document
- uuid: UUID
This fairly closely reflects the metadata currently stored by image
factory in IWHD, with the following changes:
- the current "image" type is renamed to "target_type", along with
the bucket and image reference on provider images[2]
- new image and build object types and buckets are introduced
- a build reference is added to the target image type
= Image Factory APIs =
Image factory currently supports building target images and pushing
provider images, along with the appropriate updating of image
warehouse metadata.
In order to allow the CLI to fire-and-forget, we will need to add new
build and push APIs which can handle multiple images at once.
Two alternatives to this come to mind:
* the cli could just call the existing methods sequentially for each
target passed in
* factory could provide a thin wrapper on the existing methods to do the
same thing from that side.
These would look like:
- image(image_id, template, targets[])
+ builds an image for the supplied targets
+ image_id should be ommitted, if a new image is required
If image_id _is_ passed in, is that essentially a rebuild of existing,
thereby creating an updated version (build) of the image?
+ template and targets may be ommitted if a previous build
exists and the previous template and targets will be used
- push(image_id, providers, credentials)
+ push the image the supplied providers
+ assumes factory can figure out which target image is
appropriate for each provider
+ credentials argument is a <provider_credentials/> document
Also, we'll need to rename image factory's current concept of an image
as a target image. Some initial work on that here[3].
I agree with Scott's concerns here on the multiple account per provider
thing. I don't know the answer either, so for a first pass, perhaps we
just limit it to 1 account?
= Conductor APIs =
Conductor needs APIs to support the following:
1) List the available provider types in an XML doc
2) List the available providers in an XML doc
3) Dump a <provider_credentials/> document encapsulating all of the
available provider accounts
The latter API would be subject to authentication and access control.
= Conductor IWHD Queries =
Conductor has basically two use cases for querying IWHD:
1) As a user, I want to launch a deployable:
2) As a user, I want to launch an instance of an image
For (1), conductor needs to resolve each deployable image reference to
a set of provider images. This can be done with something like:
$> $build_id = curl
http://iwhd/$image_id/latest_build
$> curl -d '$build=="'$build_id'"'
http://iwhd/target_images/_query
$> foreach $target_image_id; do curl -d
'$target_image=="'$target_image_id'"'
http://iwhd/provider_images; done
For (2), conductor needs to list all images available for launching a
standalone instance and, when the user launches the image, it needs to
list the parameters for the image. Both are easily achieved by
querying the images bucket.
= Image Building CLI =
The image building CLI is used by Aeolus users to build and upload
images from templates. It is also used by deployable authors to list
the available images.
The use cases are:
1) As an image builder or deployable author, I want to list all
images
2) As an image builder or deployable author, I want to list all
builds of an image
3) As an image builder or deployable author, I want to list all the
targets and providers an image has been built for
4) As an image builder, I want to build an image
5) As an image builder, I want to push an image to a provider
6) As an image builder, I want to import an image
7) As an image builder, I want to delete an image
8) As an image builder, I want to delete old versions of an image
9) As an image builder, I want to delete a provider image
10) As an image builder, I want to delete an image
The tool needs to interact with IWHD for listing, image factory for
building and conductor for provider/account details.
It might look like:
$> aeolus-image images # list available
images
$> aeolus-image builds $image_id # list the builds of
an image
$> aeolus-image target-images $build_id # list the target
images from a build
$> aeolus-image provider-images $target_image_id # list the provider
images from a target image
$> aeolus-image build --target ec2 --template my.tmpl # build a new image
for ec2 from the template
$> aeolus-image build --image $image_id # rebuild the image
template and targets from latest build
$> aeolus-image build --target ec2 --target rackspace \ #
In the
interest of making the cli less verbose, would it be horrible to
have target take (maybe optionally) an array? So the value is something
like:
--target [ec2,rackspace,rhevm]
--image $image_id \ #
rebuild the image with a new template and set of targets
--template my.tmpl #
$> aeolus-image push --provider ec2-us-east-1 $target_image_id # push the target
image to the specified provider
$> aeolus-image push $build_id # push all target
images for a build, to same providers as previously
$> aeolus-image push --account $provider_account $build_id # ditto, using a
specific provider account
What is the expected value of $provider_account? is that a uui or name
from conductor, or just the un/pw of actual provider account? Something
else?
$> aeolus-image push $image_id #
push all the target images for the latest build
$> aeolus-image import --provider ec2-us-east-1 $ami_id # import an AMI from
the specified provider
$> aeolus-image delete --image $image_id # deletes all builds,
target images and provider images
$> aeolus-image delete --build $build_id # deletes a build,
updating latest/parent references as appropriate
$> aeolus-image delete --target-image $target_image # deletes a target
image and its provider images
$> aeolus-image delete --provider-image $provider_image # deletes a provider
image
$> aeolus-image targets # list the values
available for the --target parameter
$> aeolus-image providers # list the values
available for the --provider parameter
$> aeolus-image accounts #
list the values available for the --account parameter
Some other notes:
- The tool will need to authenticate at least against conductor, so
--user/--pass arguments will be needed. We may also want to support
supplying these via environment variables and/or a dotfile.
- The build/push/import commands are long running, so we should
support displaying the progress of the builds and have a
--background parameter for fire-and-forget.
= Footnotes =
[1] -
https://fedorahosted.org/pipermail/aeolus-devel/2011-May/001640.html
[2] - to migrate existing data, you can do:
$> cat > update.js <<EOF
db.main.update({ _bucket : "images", object_type : "image" }, {
\$set: { _bucket : "target_images", object_type : "target_image" } },
false, true)
db.main.find({ _bucket : "provider_images", object_type :
"provider_image" }).forEach(function upd(pi) { pi.target_image=pi.image;
db.main.save(pi); })
db.main.update({ _bucket : "provider_images", object_type :
"provider_image" }, { \$unset: { image : 1 } }, false, true)
EOF
$> mongo --port 27018 repo update.js
$> mv _fs/images _fs/provider_images
This clearly only works for IWHD with the fs_autostart, but
should be easy to adjust.
[3] -
https://github.com/markmc/image_factory/commits/master
_______________________________________________
aeolus-devel mailing list
aeolus-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/aeolus-devel