--- title: Deltacloud Tracker Design (Draft) extension: html filter: - markdown - outline --- # Introduction The purpose of Deltacloud Tracker [TODO: find a snazzier name] is to allow clients to be notified of state changes in cloud resources through callbacks rather than by polling themselves. Under the hood, the tracker will of course have to poll, too. The Tracker will add a few capabilities to the Deltacloud API; this document is only concerned with changing the Deltacloud API. The other Deltacloud frontends will need to be changed accordinlgy at some point. The resource changes that will trigger a notification to clients are specific to each resource (and driver ?) They are * Instances: change to instance state * [TODO: what else ?] # API changes Tracker needs credentials for the backend cloud; it is therefore important that on each request these are set properly. In particular, Tracker will store the driver, provider, user and password each time a callback is registered. ## Registering a callback on resource creation Any collection that supports state tracking will indicate that with a feature 'state_tracking' on the corresponding create operation. This will make it possible to add the following parameters to create operations: track\_hook : the absolute URL to which to post on state changes track\_token: (optional) a security token that must be included in the callback For example, to register a callback when creating an instance, the request would look like POST /api/instances?...&track_hook=http://example.com/cb&... ## Retrieving callback details Resources that support state tracking will contain a element in their representation. The element will have the following form ## Registering a callback for an existing resource Resources that support state tracking allow updating the callback information with a PUT request to the resource. To register a callback for an instance, which will overwrite any existing callbacks, issue PUT /instances/42?track_hook=http://example.com/cb and to delete a callback, use the special token 'none': PUT /instances/42?track_hook=none # Callback When Tracker detects a change to a tracked resource, it will POST a JSON document to the hook URL [TODO: do we need XML, too ?]. The JSON body will look like { 'token': security token 'changes': [ { 'attribute': path in [JSON pointer notation](http://tools.ietf.org/html/draft-ietf-appsawg-json-pointer-00), 'old': old value, 'new': new value } ], 'resource': ... representation of resource ... } For example, for an instance that just changed from 'pending' to 'running', the callback hook would receive the following JSON document { 'token': "ABCDEFG42", 'changes': [ { 'attribute': "/state", 'old': "PENDING", 'new': "RUNNING" } ], 'resource': .. JSON object for the instance ... } The recipient for the hook should respond with 204 No Content to indicate that the update was received successfully. ## Explicitly retrieveing events Callbacks can fail, and will be retried for a while, but at some point we have to give up trying to deliver the callback (or retry so infrequently that it's not really useful to the recipient) To make it possible for recipients to catch up after a failure on their side, we'll support a 'changes' collection that only allows GET: GET /changes The response to this request will be a JSON array, where each entry is the same JSON object that is used for delivering callbacks. Note that only changes pertinent to the current provider will be delivered, i.e. clients that use Tracker to track resources in multiple providers will need to make one request for each provider. Once a change has been delivered through this mechanism, it will be considered successfully delivered. # Implementation Notes We will need to run a background job that performs the state polling. DelayedJob seems like the right tool for this; we'll want a periodic job that goes out to each backend/provider and asks for changes to tracked resources. For the first cut, we can do this resource-by-resource, but longer term we want to be more clever and use cloud-specific features (DescribeInstances for multiple instances in EC2, changed-since for OpenStack etc.) and will therefore require driver support. Conceptually, Tracker decorates the backend driver for the API operations that are modified by the 'state_tracking' feature. It is therefore tempting to implement that aspect as a Module that gets included into drivers and does the decoration. By doing this at the driver level, state tracking is immediately available to all frontends. Implementing Tracker requires that we keep state about the registered callbacks, and about the previous state of tracked resources. We'll use an RDBMS (sqlite/postgresql) and ORM (DataMapper ?) for this purpose. Very roughly, I hope we can get away with this data model (plus a jobs table): class Provider property :id, Serial property :driver, String property :provider, String property :user, String property :password, String end class Callback property :id, Serial property :hook, String property :token, String property :res_type, String property :res_id, String # Just enough to get resource from backend property :res_old, Text # Serialization of old resoure state property :last_event Timestamp # TODO: Need to track delivery state of callback and payload belongs_to :provider end ## Timings/frequencies There are a number of timings (polling frequency, how long and how often to retry change delivery) For now, they can be hardcoded, but should be in some central place so they can easily be tweaked - we do not want them controlled through the API though. We'll start with something like * Poll frequency: once a minute for instances in transient states (pending etc.), once every 15 minutes for instances in permanent states (running, stopped, ...) * Poll failure: retry at the normal frequency 5 times, then back off exponentially until frequency falls to once every two hours. [TODO: Should we issue a callback at that point ?] * Callback failure: retry callback 5 times every minute, then back off exponentially until frequency falls to once a day