Before I begin, I want to make note that the condor qmf bindings have
been pushed into the F15 updates testing repo.
I haven't messed around with this yet, but if we can subscribe to condor
events via qmf, we can just go that route.
On 06/27/2011 08:55 AM, Chris Lalancette wrote:
On 06/23/11 - 11:44:52AM, Mo Morsi wrote:
> ---
> recipes/aeolus/files/condor_config.local | 5 -----
> recipes/aeolus/manifests/conductor.pp | 4 ----
> spec/spec_helper.rb | 2 +-
> 3 files changed, 1 insertions(+), 10 deletions(-)
I thought about this a bit more this weekend, and we may want to re-think our
strategy here. Instead of removing dbomatic, I'm actually going to propose
a sort of hybrid solution.
We know that we are going to need audit logs from jobs in the future. That is,
when condor notices that a job state has changed (due to some external
influence, for instance), we want it to inform the conductor about it so that
the conductor can eventually show it to the user. In order to do this sort
of auditing, we are going to need something dbomatic-like.
Not necessarily. We can use condormatic to invoke the condor_fetch log
command to retrieve an instance event log on demand.
Alternatively if we can integrate a qmf or REST callback interface for
condor events, we could potentially get rid of dbomatic.
The biggest problem I see with dbomatic as it stands is that it writes to the
database tables that the conductor is also reading/writing from. I think we
want to get away from that. However, I'm also thinking that if we can get the
data from condor into the conductor database, it will be much easier for us
to solve the solr update-before-searching problem.
Why is multiple components accessing the db an issue?
IMO the crux of the issue is that we're representing an instance in two
different locations, in the conductor db and in condor and need to keep
those in sync.
There are a few solutions for this, one would be to derive Instance from
either ActiveResource or ActiveModel instead of ActiveRecord. The former
would require a REST frontend to condor, while the later depends on us
getting the app working against Rails 3.
Another would be to hook into the condor event process directly (going
through the event log is messier even if all the internal rotation kinks
are worked out), either via QMF or via a REST call invoked in the Condor
job update hook
http://www.cs.wisc.edu/condor/manual/v7.2/4_4Job_Hooks.html
Therefore, I propose that we keep dbomatic, but add a separate database table
for audit logs. Then every time dbomatic sees a new event, it will write it
to the audit table *only*. From conductor's point-of-view, the audit table is
read-only. When we go to do a search on something like instance state, we can
put a "pre-search" hook in so that we update the instance state from the audit
table into the normal conductor instance table, and then do the search.
I spent some time trying to implement this pre-search hook and its
tricky at best. This is because you can do something like
Instance.find(:first, :conditions => ['state = ?', 'RUNNING' ])
ActiveRecord will take the conditions, sanitize them, interpolate them,
and pass them into the SQL connection directly, eg it doesn't represent
these internally as columns before hand.
You can see this by trying to pass an invalid column name into the
conditions and see the Postgresql error, like so
Instance.find(:first, :conditions => ['foobar = ?', 'RUNNING' ])
-Mo