On 01/12/11 10:11 -0500, Matt Wagner wrote:
On Thu, Dec 1, 2011 at 8:29 AM, Francesco Vollero
<fvollero(a)redhat.com> wrote:
> Well, if I may spend my two cents here, reinventing the wheel is just unuseful at
this stage and 75% of the time we gonna have more problems than benefits.
+1 (at least) on this point.
Yes, we absolutely should use an existing library vs creating a new
one, though I did no get the impression that rolling our own was being
suggested. The other side of this though is that we did in fact
already have delayed job for background processing a few releases back
(as matt asks below). It was removed to attempt to simply our project
installation, and decrease both packaging effort and number of daemons
we require to be running. Are we sure there is not another way we can
handle any current issues without bringing such a library back in? I
am not vehemently opposed, just want to make sure we have considered
any other options before going to far down the path of once again
reintroducing something that we decided in the past we did not want.
> To have an base idea, worth to be readed carefully this link:
https://github.com/blog/542-introducing-resque since they make an analysis on the
solutions they
> used before writing Resque.
>
> Resque:
> It's been created and currently used by Github for their background processes to
substitute Delayed job
> Requirements:
> - Redis server (already in fedora)
> Video demo for Resque in Rails:
http://railscasts.com/episodes/271-resque
Redis is very cool; I worked with it a little bit on a previous
project. However, I'm not crazy about adding something like Redis just
for storing background tasks.
I have had a tab open for months, meaning to look at this, I guess now
is the time. One potential plus I see for redis (or beanstalk) right
off is that since it is a service we dont own, and separate from the
client lib that would call it in ruby, perhaps they would open up the
option of deploying a separate machine specifically for the task,
rather than forcing another process wherever conductor is run/
> Delayed Job:
> It's a quite "stable" solution, that is designed to have is natural
role in a Rails app.
>
> Video demo for dejayed job in Rails:
http://railscasts.com/episodes/171-delayed-job
At a previous employer, we used Delayed::Job. The advantage is that it
uses your existing Rails database rather than requiring something like
Redis. (For what it's worth, we ran this in a workflow where we'd
sometimes end up creating thousands of jobs in the blink of an eye, so
I was initially concerned about the impact on our database as we
hammered the table hard. It turned out to be a non-issue even with
something like 50+ jobs being processed a second.)
Am I hallucinating, or did we use Delayed Job before in Conductor?
Indeed we did, and it more or less worked fine, though I remember our
daemon for running the worker being a bit wonky. The fact that is uses
the same db may or may not be an advantage, but is sure shoudl be
considered. I am surprised it is not in fedora, this must have been a
package we were carryning in our repos, and it never got into fedora
since we dropped it.
>
> Beanstalkd:
> Fast as hell (from my point of view) simple work queue, that ship with two different
gems: stalker and beanstalk rb
>
> Video demo for Beanstalkd in Rails:
http://railscasts.com/episodes/243-beanstalkd-and-stalker
>
> Requirements:
> - beanstalkd server (already packaged in fedora)
>
> Pitfalls:
> - Stalker and beanstalk gems are not shipped with fedora, so we need to package for
ourself.
I haven't worked with (or even looked at) beanstalk before. However,
to your "pitfalls" section, it looks like _none_ of the gems mentioned
above are in Fedora -- the delayed_job and resque gems don't appear to
be packaged for Fedora yet, either.
I like that both redis and beanstalk are both in fedora, so at least
they are not more effort to package than delayed_job if we really need
one of these. Once again, I also like the idea of not having to
own/maintain the service for these 2 solutions. I'll look at the
railscasts today and provide further thoughts shoudl I have any.
Thanks for this starting point for discussion Fancesco.
-j
-- Matt