Deltacloud Tracker Design (Draft)

Jan Provazník jprovazn at redhat.com
Thu Jun 7 19:15:58 UTC 2012


Hi Lutter,
Michal pointed me to your draft of Tracker design which looks great. 
Couple of notes are inline. Sending it to aeolus-devel list too so 
others can join.

Original doc can be found here:
https://raw.github.com/lutter/deltacloud/tracker/design/site/content/tracker.mdown

> ---
> title: Deltacloud Tracker Design (Draft)
> extension: html
> filter:
>   - markdown
>   - outline
> ---
>
> # Introduction
>
> The purpose of Deltacloud Tracker [TODO: find a snazzier name] is to allow
> clients to be notified of state changes in cloud resources through
> callbacks rather than by polling themselves. Under the hood, the tracker
> will of course have to poll, too.
>
> The Tracker will add a few capabilities to the Deltacloud API; this
> document is only concerned with changing the Deltacloud API. The other
> Deltacloud frontends will need to be changed accordinlgy at some point.
>
> The resource changes that will trigger a notification to clients are
> specific to each resource (and driver ?) They are
>
> * Instances: change to instance state
> * [TODO: what else ?]

Realms are needed too. We check realms availability in Conductor, so we 
will not be able to replace our current checking tool with Tracker until 
realms are supported too.

>
> # API changes
>
> Tracker needs credentials for the backend cloud; it is therefore important
> that on each request these are set properly. In particular, Tracker will
> store the driver, provider, user and password each time a callback is
> registered.
>
> ## Registering a callback on resource creation
>
> Any collection that supports state tracking will indicate that with a
> feature 'state_tracking' on the corresponding create operation. This will
> make it possible to add the following parameters to create operations:
>
>   track\_hook : the absolute URL to which to post on state changes
>   track\_token: (optional) a security token that must be included in the callback
>
> For example, to register a callback when creating an instance, the request
> would look like
>
>     POST /api/instances?...&track_hook=http://example.com/cb&...
>
> ## Retrieving callback details
>
> Resources that support state tracking will contain a <callback/> element in
> their representation. The element will have the following form
>
>     <callback hook="http://example.com/cb">
>       <delivery status="(noevent|success|failure)" time="2012-05-23T18:23"/>
>     </callback>
>
> ## Registering a callback for an existing resource
>
> Resources that support state tracking allow updating the callback
> information with a PUT request to the resource. To register a callback for
> an instance, which will overwrite any existing callbacks, issue
>
>     PUT /instances/42?track_hook=http://example.com/cb
>
> and to delete a callback, use the special token 'none':
>
>     PUT /instances/42?track_hook=none
>

Authentication will be needed for ^ these requests to make sure that 3rd 
party doesn't modifies my callback.

> # Callback
>
> When Tracker detects a change to a tracked resource, it will POST a JSON
> document to the hook URL [TODO: do we need XML, too ?]. The JSON body will
> look like
>
>     {
>       'token': security token
>       'changes': [
>         { 'attribute': path in [JSON pointer notation](http://tools.ietf.org/html/draft-ietf-appsawg-json-pointer-00),
>           'old': old value,
>           'new': new value
>         }
>       ],
>       'resource': ... representation of resource ...
>     }
>
> For example, for an instance that just changed from 'pending' to 'running',
> the callback hook would receive the following JSON document
>
>     {
>       'token': "ABCDEFG42",
>       'changes':
>         [ { 'attribute': "/state", 'old': "PENDING", 'new': "RUNNING" } ],
>       'resource': .. JSON object for the instance ...
>     }
>
> The recipient for the hook should respond with 204 No Content to indicate
> that the update was received successfully.
>
> ## Explicitly retrieveing events
>
> Callbacks can fail, and will be retried for a while, but at some point we
> have to give up trying to deliver the callback (or retry so infrequently
> that it's not really useful to the recipient)
>
> To make it possible for recipients to catch up after a failure on their
> side, we'll support a 'changes' collection that only allows GET:
>
>     GET /changes
>
> The response to this request will be a JSON array, where each entry is the
> same JSON object that is used for delivering callbacks. Note that only
> changes pertinent to the current provider will be delivered, i.e. clients
> that use Tracker to track resources in multiple providers will need to make
> one request for each provider. Once a change has been delivered through
> this mechanism, it will be considered successfully delivered.
>

This might be a problem if Tracker is used by multiple clients - _all_ 
callbacks for a provider are fetched by one request no matter whom these 
callbacks are addressed to.

Maybe 'GET /changes' could just trigger common retry of callback 
delivery for all provider requests instead of returning changes 
directly. Another benefit is that then 'GET /changes' wouldn't have to 
be authenticated.

> # Implementation Notes
>
> We will need to run a background job that performs the state
> polling. DelayedJob seems like the right tool for this; we'll want a

+1 for delayed job - we already have this requirement in conductor

> periodic job that goes out to each backend/provider and asks for changes to
> tracked resources. For the first cut, we can do this resource-by-resource,
> but longer term we want to be more clever and use cloud-specific features
> (DescribeInstances for multiple instances in EC2, changed-since for
> OpenStack etc.) and will therefore require driver support.
>
> Conceptually, Tracker decorates the backend driver for the API operations
> that are modified by the 'state_tracking' feature. It is therefore tempting
> to implement that aspect as a Module that gets included into drivers and
> does the decoration. By doing this at the driver level, state tracking is
> immediately available to all frontends.
>
> Implementing Tracker requires that we keep state about the registered
> callbacks, and about the previous state of tracked resources. We'll use an
> RDBMS (sqlite/postgresql) and ORM (DataMapper ?) for this purpose. Very
> roughly, I hope we can get away with this data model (plus a jobs table):
>
>     class Provider
>       property :id,       Serial
>       property :driver,   String
>       property :provider, String
>       property :user,     String
>       property :password, String
>     end
>
>     class Callback
>       property :id,       Serial
>       property :hook,     String
>       property :token,    String
>       property :res_type, String
>       property :res_id,   String # Just enough to get resource from backend
>       property :res_old,  Text   # Serialization of old resoure state
>       property :last_event Timestamp
>       # TODO: Need to track delivery state of callback and payload
>       belongs_to :provider
>     end
>
> ## Timings/frequencies
>
> There are a number of timings (polling frequency, how long and how often to
> retry change delivery) For now, they can be hardcoded, but should be in
> some central place so they can easily be tweaked - we do not want them
> controlled through the API though. We'll start with something like
>
> * Poll frequency: once a minute for instances in transient states (pending
>   etc.), once every 15 minutes for instances in permanent states (running,
>   stopped, ...)
> * Poll failure: retry at the normal frequency 5 times, then back off
>   exponentially until frequency falls to once every two
>   hours. [TODO: Should we issue a callback at that point ?]
> * Callback failure: retry callback 5 times every minute, then back off
>   exponentially until frequency falls to once a day



More information about the aeolus-devel mailing list