RFC: Background Processing

Richard Su rwsu at redhat.com
Thu Apr 12 20:01:33 UTC 2012


On 04/10/2012 08:10 AM, Matt Wagner wrote:
> On Mon, Apr 09, 2012 at 05:44:45PM -0700, Richard Su wrote:
>> Hi,
>>
>> This expands on some of the notes Jan provided in other RFCs.
>> delayed_jobs and resque appears to be the most commonly deployed
>> solution.
> I have worked with delayed_job a little bit in the past, and was pretty
> happy with it.
>
>> I listed what I thought should be the requirements for a background
>> processing solution. For each requirement I then added some details
>> on how well delayed_jobs and resque could satisfy it.
>>
>> Resque contains most of the features we need. It requires Redis,
>> which is a open source project sponsored by VMware. Redis is
>> available in Fedora. But I don't see Redis available in RHEL and
>> getting it in for RHEL is the big question mark.
> Redis is really cool. But the idea of pulling it in as a dependency just
> for a queuing system feels like overkill to me. (IMHO)
>
>> https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Background_Processing
>>
>> ---
>>
>> Background Processing
>>
>> # Summary
>>
>> The two most common solutions are delayed_jobs and resque. There is a
>> good write up on github comparing other background processing
>> solutions and why they eventually steered towards delayed_jobs and
>> then resque, https://github.com/blog/542-introducing-resque.
> When we first started using DelayedJob on a previous project, I was
> really nervous about the idea of using our already-busy database for it,
> and at how it would scale as our number of jobs grew. It sounds like
> GitHub _did_ hit scaling issues, but on the project I worked on using
> DJ, we never ran into any problems even as we enqueued thousands of jobs
> and processed several hundred a minute. (Sending out customized emails,
> etc.)
>
>> The primary differences between delayed_jobs and resque are:
>>
>> At the moment, delayed_jobs doesn't have support for recurring jobs.
>> Resque does support recuring jobs through the resque-scheduler
>> extension/gem.
> I've only taken a quick look so far, but it looks like resque-scheduler
> leverages https://github.com/jmettraux/rufus-scheduler for its cron-like
> functionality. I wonder if we can take advantage of that?
>
>> resque provides a sinatra app to monitor the queue. delayed_job
>> doesn't provide monitoring tools out of the box, but we can potential
>> build something on top of rails or simply look at the contents of the
>> database table.
> So I actually view this as a plus for DJ. Resque comes with a standalone
> Sinatra app. DelayedJob can be integrated into our app cleanly by
> treating the table like it's ActiveRecord. My memory is slightly hazy,
> but I seem to recall that we set up a Job model based on DJ and set up
> some named scopes, we could easily do "Job.failed.count" or
> "Job.pending.count" for quick statistics, and we built a tiny little
> admin controller for paginating through the list of jobs.
>
>> resque requires multiple components and potentially could be more
>> difficult to support. It requries a second gem called
>> resque-scheduler. It also uses Redis as its backend and it is
>> currently not available with RHEL. This may be the deal breaker.
>>
>> # Requirements
>>
>> 1. Bucket jobs into different queues. A long running job to check
>> instance status for 1000 instances should not hold up other jobs. The
>> solution should also support multiple workers which would minimize
>> impact of longer running jobs. But using different queues will offer
>> finer grain control.
>>
>> * delayed_job: supports multiple queues through named queues starting
>> with version 3.0. Can start up multiple workers for all queues or for
>> specific queues.
>> * resque: supports multiple queues and workers.
>>
>> 2. Jobs should persist in some way. If a crash occurs, we should be
>> able to restart the system and continue with processing incomplete
>> jobs in the queue.
>>
>> * delayed_job: Jobs persists as objects stored in activerecord entries.
>> * resque: Jobs persists as json objects in redis entries. Using json
>> objects instead of actual objects which may have advanced to a
>> different version makes updating the application potentially easier.
>>
>> 3. Recurring jobs.
>>
>> * delayed_jobs: Not available, in development.
>> * resque: Through resque-scheduler extension.
>> * whenever: A potential alternative to do cron style scheduling [6].
> See also rufus-scheduler, linked above.
>
> It's not clear to me if "whenever" actually lets you define new jobs while
> running or not.
>
>> 4. Alerts. Failures should be presented to the user in some way
>> (email, conductor UI) so that appropriate actions can be taken.
>>
>> * delayed_jobs: Support code hooks for different stages in the
>> process. Hooks can be added for error, failure, success.. By default
>> workers will retry a job 25 times. We should use a lower number. No
>> sense in retrying that number of times and holding up the queue if
>> there is a hard failure somewhere in the system.  By default it also
>> deletes failed jobs, but can be configured to leave them in the queue
>> with a flag to indicate failure.
>> * resque: Failed jobs can go through additional processing using
>> different failure backends. redis, syslog, custom, etc..
>>
>> 5. A mechanism to requeue a failed job once the underlying issue has
>> been resolved. If an instance start job fails and there is a network
>> failure to a provider. Once the network is back online, we should
>> have an ability to requeue those jobs.  Not sure if this should be
>> automated or if this should be a button somewhere where a user can
>> manually requeue all or select failed jobs.
> I'm slightly uneasy about this, but perhaps I'm just not thinking it
> through fully enough. If launching an instance fails, I think, per Jan's
> robust image launching stuff, we want to just move on and try somewhere
> else, rather than having the instance potentially pop up an hour later.
> I think the general idea is good, though.
>
>> * custom
>>
>> 6. Monitor job status. We should have some way to see what is in the queue.
>>
>> * delayed_jobs: Can only view queue through activerecord database
>> entries. There is no UI so it is more difficult to see what is going
>> on.
>> * resque: Provides a sinatra app to monitor queues, jobs, and workers.
> Though as I mentioned above, I would put a different spin on it -- DJ
> lets us use our existing ActiveRecord interfaces to query this data
> easily and integrate it into our app. Resque comes with a standalone
> Sinatra app.
>
>> 7. Should not enqueue duplicate jobs.
>>
>> * custom
>>
>> 8. Ability to remove jobs from the queues and to place a pause on the
>> queues or jobs.
>>
>> * custom
>>
>> 9. Supportable in Fedora and RHEL
>>
>> * delayed_jobs: We used it in the past. Will need to carry the gem.
>> * resque: Will need to carry the gem. In addition it requires Redis
>> as the backend. Redis is available in Fedora but not in RHEL. Redis
>> is a open source project sponsored by VMware [4].
>>
>>
>> # Use Cases
>>
>> 1. Dbomatic replacement for instance and realm checking and RHEV
>> instance start.
>>
>> Each RHEV instance that is created will also lead to a job that is
>> enqueued to start that instance.
>>
>> Create a new job to perform instance status check. Create a status
>> check job for each provider account. Allow status check job to be
>> disabled/enabled per provider account.
>>
>> Create a new job to sync realms for all providers. This can be broken
>> up to a job per provider if needed.
>>
>> Create two queues. One for managing instance lifecycle. And a second
>> queue for all other jobs. Start with two workers per queue. Make the
>> number of workers configurable so that it may be adjusted when
>> needed.
> I think this is a good approach, though it bears mention that if each
> worker has a copy of the Rails runtime, this can end up gobbling up a
> lot of memory.
>
> -- Matt

Thanks for the feedback Matt.  Let's plan on going with delayed_jobs, 
unless someone dissents.

We'll need to build some additional infrastructure around delayed_jobs 
to make it more usable. First would be a way to view the contents of the 
queues from the command line, maybe in the form of a rake task or 
through aeolus-cli. I also see us adding the ability to start/stop the 
queue and to adjust the number of workers through the command line. 
Viewing the queue from Conductor would be nice, but that feels like 
follow-on work, especially if we foresee further structural changes to 
the UI.

To plug the recurring job hole in delayed_job, bringing in a scheduler 
like rufus-scheduler to queue jobs is a good idea. We'll need to think 
about how to handle alerts with recurring jobs. With dbomatic, errors 
are buried in the log. It is good that they are logged, but a person 
needs to dig around to figure out that there is a problem. If we add 
alerts, the recurring job would produce a stream of alerts/emails each 
time it failed, possibly flooding a person's inbox. We could add some 
logic to skip an alert if the same error occurred during the last run.

Speaking of alerts, is there a way to flag an alert with Conductor. I 
see there is a "Alerts" box in the Monitor overview page. How does that 
work exactly?



More information about the aeolus-devel mailing list