On 06/27/11 - 03:49:13PM, Mo Morsi wrote:
On 06/27/2011 08:55 AM, Chris Lalancette wrote:
On 06/23/11 - 11:44:52AM, Mo Morsi wrote:
recipes/aeolus/files/condor_config.local | 5 ----- recipes/aeolus/manifests/conductor.pp | 4 ---- spec/spec_helper.rb | 2 +- 3 files changed, 1 insertions(+), 10 deletions(-)
I thought about this a bit more this weekend, and we may want to re-think our strategy here. Instead of removing dbomatic, I'm actually going to propose a sort of hybrid solution.
We know that we are going to need audit logs from jobs in the future. That is, when condor notices that a job state has changed (due to some external influence, for instance), we want it to inform the conductor about it so that the conductor can eventually show it to the user. In order to do this sort of auditing, we are going to need something dbomatic-like.
Not necessarily. We can use condormatic to invoke the condor_fetch log command to retrieve an instance event log on demand.
But when do you do this? I don't think that in general you want to wait for a user action. The fact is that it could be hours between user actions, and if a lot of events happened between the user actions, then it could take a long time to fetch all of the events that happened (with the corresponding wait in the UI). It seems to me that an asynchronous process that picks up the events as they happen is the way to go here.
Alternatively if we can integrate a qmf or REST callback interface for condor events, we could potentially get rid of dbomatic.
Sure, but then you need dbomatic-qmf (that's why I mentioned something dbomatic like).
The biggest problem I see with dbomatic as it stands is that it writes to the database tables that the conductor is also reading/writing from. I think we want to get away from that. However, I'm also thinking that if we can get the data from condor into the conductor database, it will be much easier for us to solve the solr update-before-searching problem.
Why is multiple components accessing the db an issue?
IMO the crux of the issue is that we're representing an instance in two different locations, in the conductor db and in condor and need to keep those in sync.
There are a few solutions for this, one would be to derive Instance from either ActiveResource or ActiveModel instead of ActiveRecord. The former would require a REST frontend to condor, while the later depends on us getting the app working against Rails 3.
Another would be to hook into the condor event process directly (going through the event log is messier even if all the internal rotation kinks are worked out), either via QMF or via a REST call invoked in the Condor job update hook
Ah, right. That would be an interesting solution, assuming we can get all of the events we need. Then it would just be a REST message to the conductor (in the short-term), and the conductor can do the updating of both the instance state and the audit table.
Therefore, I propose that we keep dbomatic, but add a separate database table for audit logs. Then every time dbomatic sees a new event, it will write it to the audit table *only*. From conductor's point-of-view, the audit table is read-only. When we go to do a search on something like instance state, we can put a "pre-search" hook in so that we update the instance state from the audit table into the normal conductor instance table, and then do the search.
I spent some time trying to implement this pre-search hook and its tricky at best. This is because you can do something like
Instance.find(:first, :conditions => ['state = ?', 'RUNNING' ])
ActiveRecord will take the conditions, sanitize them, interpolate them, and pass them into the SQL connection directly, eg it doesn't represent these internally as columns before hand.
You can see this by trying to pass an invalid column name into the conditions and see the Postgresql error, like so
Instance.find(:first, :conditions => ['foobar = ?', 'RUNNING' ])
If we use the job hook, this goes away; we'll just update the instance field directly and then searches will just work. Definitely worth investigating.