Had a chat with Jan this morning about some of your questions. I've
replied to qs with what we talked about plus a few thoughts of my own.
On 05/24/2012 01:36 AM, David Lutterkort wrote:
We need to define in detail for each of these resources what needs to
be
tracked, and what status changes constitute an 'event', but as a general
requirement this is fine.
I suggest though that we start with something very basic, like tracking
only instance state changes, and expand from there gradually.
I'd suggest that
this is what ever changes from a->b, probably let the
driver to decide what to return since it should know what changes from
say Pending -> Running.
Do you just want a callback every time an event happens or more
detail
about what changed (like 'instance went from pending to started') ?
Same as
above
> TODO: authentication when posting data to hook url? we use oauth
between
> other components now
For the very first cut, I'd either not do any auth, or do something
incredibly cheesy like a token.
We use http basic authentication for the Conductor
API. So we can add
authentication in the callback url.
> TODO: how to handle hook failures (conductor is not accessible
and hook
> can't be invoked)?
For sure, we'd retry for a while ... after that, retry very infrequently
(like once a day) + provide an API to retrieve undelivered events. That
way, if the conductor failure is transient, everything should catch up
fairly quickly. If the conductor failure was longer (some multi-hour
maintenance event), conductor can catch up by asking for events
explicitly.
I wonder if its worth having some agreed upon policy.
For example, retry for 2 hours then revert the state change (I'm not
sure if this is even possible). But then would know if it receives
nothing in 2 hours then the request failed.
> TODO: how to handle credentials? will the stateful app keep
credentails
> permanently for each instance being checked?
As much as this worries me from a security standpoint, I don't see
another way around this - cloud API's generally don't allow any
delegation of auth.
There's a couple more TODO's connected to credentials:
TODO: how are credentials changes handled (user revokes API Key and
generates a new one) ? [not for the first cut
TODO: when are stored credentials purged ? We want to make sure we get
rid of them as quickly as possible.
Why not make this a feature. Credentials could
be stored via the API
and managed via a single login (maybe OAuth)?
Then to answer the previous question, we could add another callback on
the credentials resource that is invoked when authentication fails.
Another
I think combining these should be fairly straightforward; for
frequency
I imagine we'll build something in based on the anticipated state
change: for example, while an instance is pending, we might poll its
state pretty frequently, once it's running, we'd back off and poll much
less often.
What poll frequencies does conductor use today ?
At the moment conductor uses 60s.
It makes more sense to me though to have this set at a driver level
rather than across the board. Each provider has a different underlying
process and potentially different state machine so it makes sense to
have a different poll frequency for each, for example in EC2 you can
get the state of a bunch of instances in one request so the poll
frequencey might be higher than say a provider that only allows status
query on a per instance basis
A nice to have feature would be to then offer a high level system
setting e.g. POLL_FREQUENCY=FASTEST, SLOWEST, DEFAULT which the driver
interprets. This could be set at boot time, or on a per request basis.
Thanks
Martyn