On Thu, Aug 18, 2011 at 11:31:56AM -0500, Steve Loranz wrote:
On Aug 18, 2011, at 10:44 AM, Chris Lalancette wrote:
> On 08/17/11 - 03:46:54PM, Jason Guiditta wrote:
>> == Integration with factory (and/or 'How much metadata should iwhd
>> store?') ==
>>
>> === Part 1: Metadata ===
>> Conductor wants/will want to show the status of builds, if I understand
>> correctly. This can be broken into 3 increasingly detailed types of
>> information. Note that all these pieces of information can be gotten to
>> varying degrees from qmf events already thrown by factory:
>> * Actual Status (things like building, pushing, failed, queued, etc)
>> * Completion %
>> * What was the error, if there was a failure?
>>
>> Again, all these things are surfaced as QMF events currently, _but they
>> are captured nowhere_. So the question is, where does it make sense to
>> store these things? I only see 2 main options here, and which we choose
>> directly impacts many other aspects of this integration:
>>
>> 1. Store all this metadata in iwhd. Iwhd is the canonical home for
>> information about images, so depending how you view this, all the
>> above metrics are information about an object iwhd owns. If you follow
>> this line of reasoning, it may make sense for iwhd to store status,
>> percent complete, and error metadata.
>>
>> 2. Store these things in Conductor. In this way of thinking, Conductor
>> is trying to provide additional useful information to the user on
>> concepts iwhd does not know or care about, therefore it makes no sense
>> to have this data in iwhd.
>>
>>
>> Related to this, what happens if and when more metadata is needed about
>> any of these objects? Do we really want this information spread across
>> 2 systems?
>
> After thinking about it more, I might withdraw my previous objections to
> storing this metadata in iwhd. While it still doesn't feel exactly right from
> a layering perspective, it might be the easiest way to a solution here.
> Storing it in the conductor means we have to re-instate the model for images
> in the conductor, and then it is only used for either in-progress or failed
> images (as successful images would come from iwhd). It all starts to get a
> little messy.
>
> The short is, I wouldn't be unhappy if it went into iwhd.
I wouldn't be unhappy either, but I'm still not clear on what status is expected
in the warehouse for these objects.
Storing an image in the warehouse is the very last thing the factory does when building.
We do not upload any image or set any metadata on that image until after we have built the
image. So the only state that imagefactory would set the status attribute to would be
"COMPLETED" or "FAILED" assuming uploading to the warehouse worked or
not. If we're seeing failures on the upload step, maybe we should just be cleaning up
after ourselves.
Is this a feature, an architectural restriction, or just "the way we
currently do it"? It seems to me that if we're going to use the
warehouse as *the* datastore for this stuff, then creating the image
record in the warehouse should be the first thing the factory does
when building (even though that record might be empty). Now there's
only one place Conductor has to look to learn everything it needs to
know about the image, and there's no need for the connection between
Conductor and Factory to be persistent or have any state or anything
like that.
Is there something dramatically wrong with doing things this way?
--Hugh
Likewise, when pushing an image to a provider, storing the info about
this in warehouse is the last thing we do and again status would only be set to
"COMPLETED" or "FAILED". I think the issue remains that if we are
seeing artifacts from a failure, as was the case when deleting the temporary security
group from EC2 failed, then we need to fix those bugs and clean up after ourselves.
Setting the status to the two options above is only getting us halfway there.
>> === Part 2: Conductor <-> Factory Communication ===
>>
>> In order to build/push from Conductor, we have to bring back in the
>> dependency of Conductor communicating with factory in some way. In the
>> past, we have tried several approaches to this:
>>
>> * Embed a console directly in the webapp. This did not work well due to
>> ruby threading issues and process blocking. Also, if the console
>> encounters some kind of error, you need to restart the entire webapp to
>> recover from it.
>>
>> * Separate Process (this used the delayed_job gem). This basically had
>> a 'job' inserted into a tasks table, and the worker process would poll
>> that table for new entries. Whenever a new one was found, it would do
>> whatever was defined in the job object's execute/run type method. This
>> is a similar approach to what we used to do with taskomatic/condor, but
>> in a more generic way. As I write this, I am trying to remember if this
>> just fired off a request to connector (in next bullet), and let that
>> handle events, or something else - I cannot recall for sure which it
>> was.
>>
>> * Separate app. This was the aeolus-connector, or HTTP/QMF Bridge, as
>> it has more recently been called. It interacted with conductor via the
>> conductor's REST api (albeit simple bits) and had a console inside to
>> talk to factory. While this would require a restart if console died, it
>> could be done independently of the web app, minimizing impact.
>>
>> All of these approaches have their drawbacks, and led us to the
>> question:
>> * Is qmf actually buying us anything when talking to factory? For
>> pacemaker-cloud, we decided it would be simpler for all involved to just
>> have an HTTP interface on each side. Perhaps this would be worth
>> considering in this case as well? This may at least partly be
>> determined by weighing the amount of work for either approach, as well
>> as what actually makes the most sense from a design standpoint. One
>> thought here in favor of REST, is that I know factory was designed using
>> the delegate approach for the very reason (well, partly at least) of
>> allowing different kinds of interaction should a client want to do
>> something other than qmf.
>
> For the question about QMF, my position is still that while we may not be
> using all of the features of QMF, I still don't think it is hurting us. Most
> of the problems above are modelling problems; I think we would
> have most of the same problems if we went with a REST based approach anyway.
>
> For instance, assuming that factory had a REST frontend, when would you query
> it for build status information? On a page load, possibly? Fine, but why
> can't we just query via the QMF bus instead (as opposed to relying on events)?
It's hurting us in the sense that we're requiring users to run qpidd and install
all of the related packages in order to run the imagefactory and we're requiring the
consumers of the service to write QMF consoles; and, for all of that, we're not really
using QMF for anything that couldn't be done with more self-contained REST interface
that requires much less of the consumers.
I'm not advocating dropping the QMF interface. I think that it's a nice
deployment *option* to have and it gives users more real time status via the events. I am
advocating adding a more easily approachable option. I actually have a proposed interface
definition 90% fleshed out and will post it for review soon.
-steve
_______________________________________________
aeolus-devel mailing list
aeolus-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/aeolus-devel
--
== Hugh Brock, hbrock(a)redhat.com ==
== Engineering Manager, Cloud BU ==
== Aeolus Project: Manage virtual infrastructure across clouds. ==
==
http://aeolusproject.org ==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey