On 06/27/11 - 03:49:13PM, Mo Morsi wrote:
On 06/27/2011 08:55 AM, Chris Lalancette wrote:
> On 06/23/11 - 11:44:52AM, Mo Morsi wrote:
>> ---
>> recipes/aeolus/files/condor_config.local | 5 -----
>> recipes/aeolus/manifests/conductor.pp | 4 ----
>> spec/spec_helper.rb | 2 +-
>> 3 files changed, 1 insertions(+), 10 deletions(-)
>
> I thought about this a bit more this weekend, and we may want to re-think our
> strategy here. Instead of removing dbomatic, I'm actually going to propose
> a sort of hybrid solution.
>
> We know that we are going to need audit logs from jobs in the future. That is,
> when condor notices that a job state has changed (due to some external
> influence, for instance), we want it to inform the conductor about it so that
> the conductor can eventually show it to the user. In order to do this sort
> of auditing, we are going to need something dbomatic-like.
Not necessarily. We can use condormatic to invoke the condor_fetch log
command to retrieve an instance event log on demand.
But when do you do this? I don't think that in general you want to wait for
a user action. The fact is that it could be hours between user actions, and
if a lot of events happened between the user actions, then it could take a long
time to fetch all of the events that happened (with the corresponding wait
in the UI). It seems to me that an asynchronous process that picks up the
events as they happen is the way to go here.
Alternatively if we can integrate a qmf or REST callback interface for
condor events, we could potentially get rid of dbomatic.
Sure, but then you need dbomatic-qmf (that's why I mentioned something
dbomatic like).
>
> The biggest problem I see with dbomatic as it stands is that it writes to the
> database tables that the conductor is also reading/writing from. I think we
> want to get away from that. However, I'm also thinking that if we can get the
> data from condor into the conductor database, it will be much easier for us
> to solve the solr update-before-searching problem.
Why is multiple components accessing the db an issue?
IMO the crux of the issue is that we're representing an instance in two
different locations, in the conductor db and in condor and need to keep
those in sync.
There are a few solutions for this, one would be to derive Instance from
either ActiveResource or ActiveModel instead of ActiveRecord. The former
would require a REST frontend to condor, while the later depends on us
getting the app working against Rails 3.
Another would be to hook into the condor event process directly (going
through the event log is messier even if all the internal rotation kinks
are worked out), either via QMF or via a REST call invoked in the Condor
job update hook
http://www.cs.wisc.edu/condor/manual/v7.2/4_4Job_Hooks.html
Ah, right. That would be an interesting solution, assuming we can get all of
the events we need. Then it would just be a REST message to the conductor
(in the short-term), and the conductor can do the updating of both the instance
state and the audit table.
>
> Therefore, I propose that we keep dbomatic, but add a separate database table
> for audit logs. Then every time dbomatic sees a new event, it will write it
> to the audit table *only*. From conductor's point-of-view, the audit table is
> read-only. When we go to do a search on something like instance state, we can
> put a "pre-search" hook in so that we update the instance state from the
audit
> table into the normal conductor instance table, and then do the search.
I spent some time trying to implement this pre-search hook and its
tricky at best. This is because you can do something like
Instance.find(:first, :conditions => ['state = ?', 'RUNNING' ])
ActiveRecord will take the conditions, sanitize them, interpolate them,
and pass them into the SQL connection directly, eg it doesn't represent
these internally as columns before hand.
You can see this by trying to pass an invalid column name into the
conditions and see the Postgresql error, like so
Instance.find(:first, :conditions => ['foobar = ?', 'RUNNING' ])
If we use the job hook, this goes away; we'll just update the instance field
directly and then searches will just work. Definitely worth investigating.
--
Chris Lalancette