Background Jobs in conductor

Francesco Vollero fvollero at redhat.com
Thu Dec 1 16:13:36 UTC 2011


On Thu, Dec 01, 2011 at 10:55:18AM -0500, Jason Guiditta wrote:
> On 01/12/11 10:11 -0500, Matt Wagner wrote:

[snip]
> 

My idea is to create few sample applications (in sinatra i can do it now) and rate them with few parameters that we think are really important for our project.

What's your thoughts on that?

(follow down)

> >>Resque:
> >> It's been created and currently used by Github for their background processes to substitute Delayed job
> >> Requirements:
> >> - Redis server (already in fedora)
> >> Video demo for Resque in Rails: http://railscasts.com/episodes/271-resque
> >
> >Redis is very cool; I worked with it a little bit on a previous
> >project. However, I'm not crazy about adding something like Redis just
> >for storing background tasks.
> >
> 
> I have had a tab open for months, meaning to look at this, I guess now
> is the time.  One potential plus I see for redis (or beanstalk) right
> off is that since it is a service we dont own, and separate from the
> client lib that would call it in ruby, perhaps they would open up the
> option of deploying a separate machine specifically for the task,
> rather than forcing another process wherever conductor is run/
> 

Me too, i started playing around for another project and i liked the way it work with normal/sometimes intense load.


> >>Delayed Job:
> >> It's a quite "stable" solution, that is designed to have is natural role in a Rails app.
> >>
> >> Video demo for dejayed job in Rails: http://railscasts.com/episodes/171-delayed-job
> >
> >At a previous employer, we used Delayed::Job. The advantage is that it
> >uses your existing Rails database rather than requiring something like
> >Redis. (For what it's worth, we ran this in a workflow where we'd
> >sometimes end up creating thousands of jobs in the blink of an eye, so
> >I was initially concerned about the impact on our database as we
> >hammered the table hard. It turned out to be a non-issue even with
> >something like 50+ jobs being processed a second.)
> >
> >Am I hallucinating, or did we use Delayed Job before in Conductor?
> >
> Indeed we did, and it more or less worked fine, though I remember our
> daemon for running the worker being a bit wonky. The fact that is uses
> the same db may or may not be an advantage, but is sure shoudl be
> considered.  I am surprised it is not in fedora, this must have been a
> package we were carryning in our repos, and it never got into fedora
> since we dropped it.

I vote to avoid it as well if possible.

> >>
> >>Beanstalkd:
> >> Fast as hell (from my point of view) simple work queue, that ship with two different gems: stalker and beanstalk rb
> >>
> >> Video demo for Beanstalkd in Rails: http://railscasts.com/episodes/243-beanstalkd-and-stalker
> >>
> >> Requirements:
> >> - beanstalkd server (already packaged in fedora)
> >>
> >> Pitfalls:
> >> - Stalker and beanstalk gems are not shipped with fedora, so we need to package for ourself.
> >
> >I haven't worked with (or even looked at) beanstalk before. However,
> >to your "pitfalls" section, it looks like _none_ of the gems mentioned
> >above are in Fedora -- the delayed_job and resque gems don't appear to
> >be packaged for Fedora yet, either.
> >
> 
> I like that both redis and beanstalk are both in fedora, so at least
> they are not more effort to package than delayed_job if we really need
> one of these.  Once again, I also like the idea of not having to
> own/maintain the service for these 2 solutions.  I'll look at the
> railscasts today and provide further thoughts shoudl I have any.
> Thanks for this starting point for discussion Fancesco.
> 

Well, i tried beanstalkd and i liked it much, very skinny and good looking. Like Resque.


Francesco

> -j
> 
> >-- Matt



More information about the aeolus-devel mailing list