Background Jobs in conductor

Matt Wagner mwaggy at gmail.com
Thu Dec 1 15:11:42 UTC 2011


On Thu, Dec 1, 2011 at 8:29 AM, Francesco Vollero <fvollero at redhat.com> wrote:
> Well, if I may spend my two cents here, reinventing the wheel is just unuseful at this stage and 75% of the time we gonna have more problems than benefits.

+1 (at least) on this point.

> To have an base idea, worth to be readed carefully this link: https://github.com/blog/542-introducing-resque since they make an analysis on the solutions they
> used before writing Resque.
>
> Resque:
>  It's been created and currently used by Github for their background processes to substitute Delayed job
>  Requirements:
>  - Redis server (already in fedora)
>  Video demo for Resque in Rails: http://railscasts.com/episodes/271-resque

Redis is very cool; I worked with it a little bit on a previous
project. However, I'm not crazy about adding something like Redis just
for storing background tasks.

> Delayed Job:
>  It's a quite "stable" solution, that is designed to have is natural role in a Rails app.
>
>  Video demo for dejayed job in Rails: http://railscasts.com/episodes/171-delayed-job

At a previous employer, we used Delayed::Job. The advantage is that it
uses your existing Rails database rather than requiring something like
Redis. (For what it's worth, we ran this in a workflow where we'd
sometimes end up creating thousands of jobs in the blink of an eye, so
I was initially concerned about the impact on our database as we
hammered the table hard. It turned out to be a non-issue even with
something like 50+ jobs being processed a second.)

Am I hallucinating, or did we use Delayed Job before in Conductor?

>
> Beanstalkd:
>  Fast as hell (from my point of view) simple work queue, that ship with two different gems: stalker and beanstalk rb
>
>  Video demo for Beanstalkd in Rails: http://railscasts.com/episodes/243-beanstalkd-and-stalker
>
>  Requirements:
>  - beanstalkd server (already packaged in fedora)
>
>  Pitfalls:
>  - Stalker and beanstalk gems are not shipped with fedora, so we need to package for ourself.

I haven't worked with (or even looked at) beanstalk before. However,
to your "pitfalls" section, it looks like _none_ of the gems mentioned
above are in Fedora -- the delayed_job and resque gems don't appear to
be packaged for Fedora yet, either.

-- Matt



More information about the aeolus-devel mailing list