Hi Jan,
On Wed, 2012-05-23 at 09:13 +0200, Jan Provaznik wrote:
Deltacloud is going to implement a stateful app on top of DC API.
This
app will do very similar job to what we planned with Cloud State. Some
basic ideas/thoughts about Cloud State are described on wiki:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Cloud_State
So for us it would be much easier to use this deltacloud app instead of
implementing the same thing.
I think this would be a very useful addition to DC, as an optional
add-on (meaning you can either use DC as a strict proxy as you do today,
or add state tracking, with some additional infrastructure, like a DB)
Architecturally, this seems to me like something we'd want to hook into
the existing drivers - essentially have a state tracking driver that
decorates the actual backend driver (though I have to think this through
in a little more detail)
A DC+state tracking server would expose only the 'normal' IaaS API as
much as possible, with a few additions to control state tracking. For
the DC API it's pretty clear how we can expose these additions, for CIMI
somewhat clear, and for EC2 it's still very murky ;) But I assume your
immediate need is for the 'classic' DC API.
Here are some basic requirements which we would need on Conductor
side
from the stateful app:
1) the app should check state of not only instances but also other
resources - we need especially realms because we check availability status
We need to define in detail for each of these resources what needs to be
tracked, and what status changes constitute an 'event', but as a general
requirement this is fine.
I suggest though that we start with something very basic, like tracking
only instance state changes, and expand from there gradually.
2) it should be possible to register hooks for a resource I want to
watch, hook could be just an URL, when the resource is changed data are
POSTed to the registered URL.
Do you just want a callback every time an event happens or more detail
about what changed (like 'instance went from pending to started') ?
TODO: authentication when posting data to hook url? we use oauth
between
other components now
For the very first cut, I'd either not do any auth, or do something
incredibly cheesy like a token.
TODO: how to handle hook failures (conductor is not accessible and
hook
can't be invoked)?
For sure, we'd retry for a while ... after that, retry very infrequently
(like once a day) + provide an API to retrieve undelivered events. That
way, if the conductor failure is transient, everything should catch up
fairly quickly. If the conductor failure was longer (some multi-hour
maintenance event), conductor can catch up by asking for events
explicitly.
TODO: how to handle credentials? will the stateful app keep
credentails
permanently for each instance being checked?
As much as this worries me from a security standpoint, I don't see
another way around this - cloud API's generally don't allow any
delegation of auth.
There's a couple more TODO's connected to credentials:
TODO: how are credentials changes handled (user revokes API Key and
generates a new one) ? [not for the first cut]
TODO: when are stored credentials purged ? We want to make sure we get
rid of them as quickly as possible.
3) frequency of instance status checking: we probably don't need
to have
this configurable, but the checking frequency should be as low as
possible - various actions are hooked to instance state changes (for
example runtime is measured by this). It should be also scalable - if
there are 100 instances for one provider account, we still need check
all of them. Would be nice to use multiple parallel requests to provider
if possible or checking interval might be dynamically increased
according to the actual length of queue).
I think combining these should be fairly straightforward; for frequency
I imagine we'll build something in based on the anticipated state
change: for example, while an instance is pending, we might poll its
state pretty frequently, once it's running, we'd back off and poll much
less often.
What poll frequencies does conductor use today ?
David