On Nov 1, 2010, at 7:52 PM, Ian Main wrote:
On Mon, 2010-11-01 at 10:07 -0400, Chris Lalancette wrote:
On 11/01/10 - 02:32:02PM, Ladislav Martincik wrote:
Hi Chris,
just curious, why do you think that file observation is good solution?
I guess the question is, why do you think it is not? If you were to do something like pipes, that would just be a file anyway, so you wouldn't really get much efficiency gains.
Ok. Let's assume that file storage is a good solution. How do u assure that: 1) When the file gets big it will not blow your memory consumption 2) What if somebody deletes the file by accident or change its content (backup, logrotate) 3) What if you miss some messages? Fault tolerant.
Now, I agree, we eventually want to switch to QMF; that will allow us to do it over the network, meaning we can have condor running on a different machine from the aggregator. But at the moment I don't really see a compelling need to move to anything but QMF.
I think the discussion should about Condor being the background job processor. IMHO choosing simple DelayedJob gem/plugin would make things much more simpler and faster to implement from beginning.
Having sufficient message queue would make things even much better.
I agree, there's nothing wrong with the file observation method we are using now. The format is good (XML) and we can use eg inotify or other mechanism (kqueue on bsd) to be notified of new data.
AFAIK QMF is still not prime time for condor, though I should poke in on that front again now as it's been some time.
Ian
My main goal for sub. of rb-inotify has been the fact that I do most of the dev. work on my MacBook and having to comment all the time rb-inotify to be able to run server is just painful.
-- Ladislav