Background Jobs in conductor

Kenneth Keiter kenkeiter at redhat.com
Thu Dec 1 17:00:34 UTC 2011


My notes inline...

On Dec 1, 2011, at 7:11 AM, Matt Wagner wrote:

> On Thu, Dec 1, 2011 at 8:29 AM, Francesco Vollero <fvollero at redhat.com> wrote:
>> Well, if I may spend my two cents here, reinventing the wheel is just unuseful at this stage and 75% of the time we gonna have more problems than benefits.
> 
> +1 (at least) on this point.

This would be ridiculous -- there are innumerable background worker tools out there; many are designed for Rails.

> 
>> To have an base idea, worth to be readed carefully this link: https://github.com/blog/542-introducing-resque since they make an analysis on the solutions they
>> used before writing Resque.
>> 
>> Resque:
>> It's been created and currently used by Github for their background processes to substitute Delayed job
>> Requirements:
>> - Redis server (already in fedora)
>> Video demo for Resque in Rails: http://railscasts.com/episodes/271-resque
> 
> Redis is very cool; I worked with it a little bit on a previous
> project. However, I'm not crazy about adding something like Redis just
> for storing background tasks.

Resque is awesome, and pretty light-weight -- I've used it myself. It provides a nice web interface and API which you can use to manage background tasks. I'm not crazy about the Redis dependency, either. I think it might be overkill

> 
>> Delayed Job:
>> It's a quite "stable" solution, that is designed to have is natural role in a Rails app.
>> 
>> Video demo for dejayed job in Rails: http://railscasts.com/episodes/171-delayed-job
> 
> At a previous employer, we used Delayed::Job. The advantage is that it
> uses your existing Rails database rather than requiring something like
> Redis. (For what it's worth, we ran this in a workflow where we'd
> sometimes end up creating thousands of jobs in the blink of an eye, so
> I was initially concerned about the impact on our database as we
> hammered the table hard. It turned out to be a non-issue even with
> something like 50+ jobs being processed a second.)
> 

Delayed job is probably the best solution for this task. Lightweight, no additional dependencies, uses our existing database, and is MORE than fast enough to accomplish what we need to accomplish.

> Am I hallucinating, or did we use Delayed Job before in Conductor?
> 

You're not hallucinating.. We did. No idea what happened to it...

>> 
>> Beanstalkd:
>> Fast as hell (from my point of view) simple work queue, that ship with two different gems: stalker and beanstalk rb

Is speed a requirement at the moment? Because if it's not, that eliminates the primary reason to use this. The added packaging work makes this impractical, as well. Beanstalk is awesome (have used it before); however, much like a NoSQL database (or really, any engineering solution), you need to take a hard look at your problem domain before you go this route.

>> 
>> Video demo for Beanstalkd in Rails: http://railscasts.com/episodes/243-beanstalkd-and-stalker
>> 
>> Requirements:
>> - beanstalkd server (already packaged in fedora)
>> 
>> Pitfalls:
>> - Stalker and beanstalk gems are not shipped with fedora, so we need to package for ourself.
> 
> I haven't worked with (or even looked at) beanstalk before. However,
> to your "pitfalls" section, it looks like _none_ of the gems mentioned
> above are in Fedora -- the delayed_job and resque gems don't appear to
> be packaged for Fedora yet, either.
> 
> -- Matt

--
Kenneth Keiter
Sr. Software Engineer, Cloud BU
kenkeiter at redhat.com
http://twitter.com/kenkeiter
IRC: tigger or tiggerer






More information about the aeolus-devel mailing list