Hi Ian,
On Tue, 2011-05-24 at 13:12 -0500, Ian McLeod wrote:
Mark,
Current work demands prevent me from drafting a full point-by-point
reply right now. So, I'll make a few observations:
* Because it represents the actual contents of an image, ICICLE cannot
be generated for Target Image objects for snapshot style builds/clouds.
For these clouds the Target Image object (what we used to call simply an
Image) is a pure placeholder and an actual "realized" image is only
created when you "push" to create a Provider Image.
In other words, in some cases "building" (or the existing "image()"
API
call) establishes the final image content while in others "pushing"
establishes it.
We discussed this once several months ago. At the time we tentatively
agreed that Images (old definition - new term being Target Images) _may_
contain ICICLE but only Provider Images _must_ contain ICICLE.
So the idea that, with snapshot builds, we will generate the icicle in
the EC2 reference during the push stage.
Why do the upload builds in FedoraBuilder not generate the icicle in the
build stage and copy the reference across in the upload stage? i.e. at
the moment, all provider image icicle references are set to 'none' by
FedoraBuilder
Also, I don't actually see image factory generating an icicle anywhere.
e.g. I don't see Oz's generate_icicle() being called anywhere
Also, I assume the "*must* contain an icicle" doesn't apply to imported
images?
* Doing builds one after the other doesn't actually guarantee
that they
are identical due to the dynamic nature of yum repos (or the SMB shares
or HTTP servers containing package content for Windows). As a result,
guaranteeing the cross-target and cross-provider consistency implied by
the "Image Build" object will require some non-trivial additional work
on the Factory side and may not be possible in the general case.
We will likely have to explicitly verify, after building, that the
resulting Target Images and/or Provider Images are, roughly speaking,
equivalent.
An example to illustrate some of the concerns above:
Amazon EC2 recently added a second Asia-Pacific region. (This is
actually true.)
We support both upload style and snapshot style building of Images on
EC2.
Users with "upload style" EC2 builds can simply "push" their
existing
Target Image builds to the new region and will be guaranteed that they
are identical. (And the new Provider Image can actually reference a
shared ICICLE object inherited from the Target Image.)
Users with "snapshot style" EC2 builds who "push" to the new
Provider/Region will repeat the "modify in place and snapshot" step. If
their template has, for example, the "fedora-updates" repo in it, they
will almost certainly end up with newer package content in the Provider
Image.
It's not clear to me what we should do in this case. We can,
potentially, verify that the new snapshot Provider Image is different.
Should we then automatically create new object versions down the chain?
(That is, new Target Image and Image Build objects.)
Instinctively, I appreciate that guaranteeing the images from a build
are identical is difficult.
But, in your example, I'm not sure I follow the problem. Assuming that
all Fedora mirrors are in sync, is it the time difference between the
build and push stages? i.e if you built your upload image a week ago and
then pushed your upload image and snapshot image today, you're likely to
have a different set of packages in each image?
How about if the image building tool didn't expose the two stages? i.e.
you do the build and push in one go, reducing the race condition. If you
want to push to a new provider, you just go through both build/push
again
The key thing we're trying to do here is get to a stage where a user can
launch an Image (or Deployable that references the Image) and be
confident that they'll get an equivalent image no matter which provider
is chosen
It sounds like if we expose the build/push stages separately, the user
can walk themselves into this race condition. So the answer might be to
discourage that?
i.e. yes, we may be unhappy that just because a user wants to push to a
new EC2 region, she also has to rebuild and upload a new RHEV-M image.
That might appear wasteful, but if it's the only way the user can be
confident that the bits are identical .... ?
(This is distinct from the capabilities of image factory itself. It can
still have the two stages, but the workflow we'd encourage people to
follow with aeolus-image wouldn't expose it)
Cheers,
Mark.