On 07/27/2011 10:21 PM, Matt Wagner wrote:
Hi Tomas,
This is an excellent write-up. As a result, most of my notes below are
tangential musings. :)
On Wed, Jul 27, 2011 at 08:50:29PM +0200, Tomas Sedovic wrote:
> Most of the functionality is already in place: when an instance crashes
> or is stopped outside of Conductor, Condor picks it up and updates the
> status accordingly.
It occurs to me that right now, much of this happens as dbomatic tails
Condor's EventLog and then directly updates the database. We never really
liked this, but there was no better way to do it.
If we're going to be providing an API on top of this, perhaps we can
finally make dbomatic use an API instead of directly modifying the
database. I'm not sure this is in scope for this iteration, though.
This occurred to me as well, but I think it deserves its own thread.
My thinking here is this:
1) we keep Condor/dbomatic as is for now. It ain't ideal, but we got it
to work. And we'll add the dead-simple API for other services to notify
us about additional changes. That will be for notifications only -- no
side effects should be taken as a result.
We plan to have API for controlling conductor, so if someone wants to
make a persistent change, they can use that.
This will be very easy to implement with little risk of breakage.
2) Sooner or later, we will need a good flexible way of doing
asynchronous operations.
Something where you can say "go to that URL that takes five minutes to
load and lemme know when you're done" without blocking the app.
Right now everything except for operating/monitoring backend instances
calls uses blocking code. Condor takes care of the Deltacloud calls but
does nothing else.
Given how many different services this thing will consist of, spread
possibly across networks with various latencies, we will need some way
to do async operations.
I think we can use Condor for that but we'll have to change the way we
send jobs to it and the way we get the results from.
So that needs to happen and once we get to it we should probably make
sure it's done in a robust way that we're comfortable with. And then we
can use it for everything -- including what dbomatic does now.
> Doing this consistently across the app would be great but isn't trivial.
> If we just hacked a JavaScript snipet for every time we needed this, it
> would soon became a mess. Libraries like Backbone.js[1] should help, but
> it takes some time and effort getting them in. Backbone in particular
> isn't a drop-in solution. Others may be. Suggestions?
We had talked a while about starting to use Backbone. Did that not
happen? (I'm completely impartial here, just curious.)
It did not happen yet. I've spent some time with it, but it wasn't
really a drop-in solution. We were close to the release then and the
risk of breaking the UI in all sorts of subtle ways was too high.
We do want to use something like that eventually because it will
increase the responsiveness of the app.
If it makes sense, we can add it in this iteration. However, I'm
beginning to have doubts that Backbone is ideal for Conductor, so there
may be another discussion about that once we start thinking about
implementing the ajaxified UI.
> * Better UI
> - Right now we show instance's history in the ugly& forgotten
> instance#show page. It would make sense to make it look prettier, and
> possibly display the events on the Deployment page for all its
> associated instances.
It's a little bit of a tangent, but it'd be really swell if we could
either brush up the Instances page and make it a first-class citizen
again, or move the requisite functionality into something else in the
new UI. Right now we find the instances page handy but there's no way of
getting there.
Yup. It makes sense to display the info there. But I think we should put
it into the deployments details page rather than creating whole another
UI area with lots of tabs, buttons, etc.
If I understand it correctly, the current goal is to view present
deployments as the atomic unit that the users work with inside Conductor.
Since they are not atomic by their nature, we do have to display the
information about individual instances, but we want the users to think
in terms of deployments, not instances.
Whether that goal still makes sense or not is a different question, but
I think that's what the UX people should decide by talking to and
observing our users.
> * Make the event log persistent
>
> When an instance gets deleted now, all events get deleted with it. What
> we want to do instead is to hide the instance from the UI but keep it
> and the events around so that we can still generate correct reports.
>
> Matt Wagner suggested a couple of gems[2][3] that should handle this
> pretty transparently. They're worth looking into as doing this by
> ourselves could get tricky.
I've used acts_as_paranoid and acts_as_trashable on a few projects in
the past. acts_as_paranoid adds a deleted_at and overrides AR's .find,
and seems to work flawlessly. (You can do find_with_deleted or
find_only_deleted when you actually want to bring back deleted records.)
acts_as_trashable actually deletes the row from the database, but
dumps the object as YAML or something and stuffs it into a "trash"
table for archival.
They both sound great. I would prefer starting with acts_as_paranoid as
it sounds easier to use for generating reports.
I fear it may mean a performance hit (I've no experience with that,
someone who has, please speak up), but since we don't really lose any
data, we can always switch to acts_as_trashable later on and just run a
migration.
We could probably go the other way around as well, but it sounds more
difficult.
Please correct me if I'm wrong with that regard or if there are
compelling reasons not to use acts_as_paranoid from the beginning.
> Generating And Exporting Status Reports
> =======================================
>
> Right now, this will mean creating a simple CSV file that can be downloaded.
>
> Possibly something like:
>
> GET /conductor/reports
It would be nifty to implement this in a respond_to block:
respond_to do |format|
format.html { ... }
format.csv { ... }
end
Then we can have /conductor/reports/foo give a nice HTML table, and
/conductor/reports/foo.csv serve up the CSV file.
Excellent point. I agree completely.
> Features& Tasks
> ================
> ... snip ...
> * Periodically update uptime values so that they're always semi up-to-date
We could actually just periodically sync a timer on the page, and use
that on-page control to keep the timers running continually. I'm not
sure if having the times increment every second would be more useful
than distracting, though.
Yeah, that would be distracting. But I was thinking that updating it,
say, every minute or five would be great.
I'm imagining a scenario when you launch a massive deployment and while
the page still shows everything in pending, you leave the page up and go
to lunch.
When you get back, it shows everything as running/error and an uptime
that is not horribly behind.
> As an admin, I want to get the full reports even when the instances have
> been deleted
> * Make the event logs persistent even on instance deletion
> * Preserve the data associated with the deleted object for events/reports
How far through associations do we want to go? If I launch an instance,
stop it, and then delete the deployment it was in when I'm done, should
we save the instance record? The deployment? The deployable XML? (I
suspect the answer is that we want to save everything, but thought I'd
ask to make sure.)
Yeah, I think we want to save everything unless there's a compelling
reason not to.
When we do have all the data, we can always delete or archive it later,
should we need to. But if we don't have the data and realize that we
want them, it's too late.
Thanks for your great comments and suggestions!
Overall, this looks great!
-- Matt
_______________________________________________
aeolus-devel mailing list
aeolus-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/aeolus-devel