On 07/28/2011 09:47 AM, Tomas Sedovic wrote:
On 07/28/2011 01:02 PM, Steven Dake wrote:
> Tomas,
>
> good work - couple refinements inline.
>
> On 07/27/2011 11:50 AM, Tomas Sedovic wrote:
>> Hey everyone,
>>
>> One of the proposed features for this iteration is Status Reporting.
>> There are four main components to it:
> <snip>
>
>
>> Matahari And Other External Services
>> ====================================
>>
>> Again, this is outside of this iteration's scope.
>>
>
> I disagree - if we leave this outside of this iteration's scope we have
> 3 nearly full-time engineers blocked not doing any work. We are ready
> to start tackling these integration issues now.
That's great. I wasn't aware that we have people who can help us with
this (I have to shamefully admit that I've only managed to read the HA
thread earlier today).
>
>> As we'll want to get notification from Matahari (and maybe other
>> services) we'll need a way to get notified about events that should be
>> logged but that Conductor itself cannot detect.
>>
>> From talking to Matahari folks, it seems they need Conductor to let
>> them know when an instance was started and stopped.
>>
>> If I understand it correctly, the whole notion of Config Server is to
>> have something that communicates with the in-instance agents (matahari).
>> Thus, we may want to hijack that and send the messages through Config
>> Server rather than directly.
>>
>> This will save us the trouble of creating yet another communication
>> interface and discovering the matahari agents. Rather, we'll piggyback
>> on something that's already planned anyway.
>>
>
> Managing Matahari discovery and state is in the domain of pacemaker
> cloud. If Aeolus integrates with Matahari directly, developers will be
> reimplementing pacemaker-cloud in some fashion and spend several months
> doing it.
Makes sense. Code that doesn't have to go into Conductor is a good code.
I wasn't that familiar with pacemaker cloud but I was hoping we could
leverage something like that.
I was thinking about the Config Server, which will be communicating with
the instances, but if Pacemaker Cloud is designed for this, great.
>
>> As for other services notifying Conductor, this can be as simple as:
>>
>> POST /conductor/instances/3141/events
>>
>> So, like, fyi, that thingy over over here has changed.
>>
>> Possibly with a timestamp in the request body so that we can log when
>> the event happened vs. when it was received.
>>
>> However, this must be tied into whatever way we implement cross-module
>> authentication and authorization, otherwise everyone will be able to
>> create phoney events.
>>
>
> This is one advantage of using QMF for events - authentication is free
> in this model.
>
> Is QMF a nonstarter for aeolus?
Given the nature of Conductor (a Rails webapp), it feels more natural to
communicate via HTTP APIs whenever possible.
That said, if it would make more sense to use QMF, we could manage.
Personally, I have zero experience with QMF, but I know some Conductor
folks do.
Jay, could you share your thoughts, on this?
>
>>
>> Features& Tasks
>> ================
>>
>> As a user, I want to see the uptime of my instances and deployments
>> * Display uptime of instances in the UI
>> * Fix the uptime calculation for deployments
>>
>> As a user, I want to see Conductor always show the current status
>> without having to press<F5> every time
>> * Implement Backbone.js or some other system that will make this easy for us
>> * Dynamically update the instance/deployment status across Conductor
>> * Dynamically update the scoreboards in the Monitor section
>> * Dynamically show/hide errors as they unfold happen on the backend
>> * Periodically update uptime values so that they're always semi up-to-date
>>
>> As an admin, I want to see a report of everything that happened to instances
>> * Generate a report of all the Events in a machine-processable format
>> * Add a download link for the report into the UI
>> * Make this available to administrators only
>>
>> As an admin, I want to get the full reports even when the instances have
>> been deleted
>> * Make the event logs persistent even on instance deletion
>> * Preserve the data associated with the deleted object for events/reports
>>
>> As an admin, I want to be able to query the status report to get to the
>> info I need
>> - note: this is probably out of scope for the Iteration 4.
>> * Filter by a specific timeframe
>> * Filter by a specific user
>> * Filter by a specific pool
>>
>> As a user, I want to be notified on changes that happen inside of an
>> instance
>> - note: this is out of scope for the Iteration 4.
>> * Implement a way of communicating with Matahari
>> * Provide an API for getting event notifications from external services
>>
>> As a user, I want to be able to see and query events of objects that I
>> have access to
>> - note: this is out of scope for the Iteration 4.
>> * Display the status reports filtered by user's permissions
>>
>
> Please see our commitments for this iteration:
>
https://fedorahosted.org/pipermail/aeolus-devel/2011-July/003602.html
Thanks for the comments, Steven.
We should definitely discuss this further. I'll try to read up more on
Pacemaker Cloud and Matahari tomorrow.
In the meantime, what do you see as the main things that need to happen
so that the reporting flows between Conductor and the instances it manages?
That is:
* What (if any) data should conductor be sending to pacemaker
pacemaker-cloud needs the deployable and assembly information,
preferably in XML format. Prior to launching a deployable, aeolus would
tell pacemaker-cloud about it. Today this happens with a QMF call into
our system. We are not locked into QMF at this interface - this is only
what our prototype provides today.
* What will pacemaker send back to conductor
pacemaker-cloud generates QMF events when state changes occur in
deployables or assemblies. We currently have a format for these events,
but are open to formatting changes. We are not locked into QMF at this
interface and open to protocol changes.
* What protocol and format does pacemaker use. QMF?
yes - matahari requires QMF and that isn't likely to change in the long
term. In terms of our "external interfaces", they are currently
implemented using QMF and XML but QMF could be changed to something more
rails friendly if required.
* Suppose that pacemaker crashes or something. Conductor will then be
cut off from some of instances' information. Would it make sense for
conductor to talk to Matahari directly, then? If so, could we reuse the
pacemaker cloud code not to duplicate the effort?
suppose conductor fails....
All software fails, but precautions have been taken to protect this
component:
+ The code has recovery to protect from unplanned stop failures (such
as a crash).
+ The developers have applied high availability development
principles to maximize MTBF. See slide 14:
http://www.redhat.com/summit/2011/presentations/summit/whats_ne/thursday/...
+ The project code footprint is extremely small (python/sh are
project test cases):
Totals grouped by language (dominant language first):
cpp: 2432 (46.26%)
sh: 1124 (21.38%)
python: 975 (18.55%)
ansic: 726 (13.81%)
Total Physical Source Lines of Code (SLOC) = 5,257
compare for yourself to code footprint of other components (yum install
sloccount; sloccount projectdir).
The way to reuse pacemaker cloud codebase is to integrate with it.
Communicating with matahari via conductor solves a subsset of problems
solved by pacemaker cloud.
* Anything else I'm missing?
shutdown and diagnostics.
Key tasks:
+ aeolus sends deployable and assembly information to pacemaker-cloud
prior to launching an instance.
+ constraint that assembly UUIDs given to this call match the
Matahari instance id for the assembly.
+ aeolus sends deployable shutdown information to pacemaker-cloud
+ agree on event format and feedback API.
+ implement feedback API.
+ document diagnostic capabilities.
+ sort out image building to include Matahari including uuid and qpid
authentication credentials.
> Thomas
>>
>>>
>>> [1]:
http://documentcloud.github.com/backbone/
>>> [2]:
https://github.com/bdurand/acts_as_trashable
>>> [3]:
https://github.com/technoweenie/acts_as_paranoid
>>> _______________________________________________
>>> aeolus-devel mailing list
>>> aeolus-devel(a)lists.fedorahosted.org
>>>
https://fedorahosted.org/mailman/listinfo/aeolus-devel
>>
>> _______________________________________________
>> aeolus-devel mailing list
>> aeolus-devel(a)lists.fedorahosted.org
>>
https://fedorahosted.org/mailman/listinfo/aeolus-devel
> _______________________________________________
> aeolus-devel mailing list
> aeolus-devel(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/aeolus-devel