RFC: Background Processing

Richard Su rwsu at redhat.com
Fri Apr 13 20:22:44 UTC 2012


On 04/13/2012 03:58 AM, Jan Provazník wrote:
> On 04/12/2012 10:41 PM, Matt Wagner wrote:
>> On Thu, Apr 12, 2012 at 01:01:33PM -0700, Richard Su wrote:
>>>
>>> Thanks for the feedback Matt.  Let's plan on going with delayed_jobs,
>>> unless someone dissents.
>>>
>>> We'll need to build some additional infrastructure around
>>> delayed_jobs to make it more usable. First would be a way to view the
>>> contents of the queues from the command line, maybe in the form of a
>>> rake task or through aeolus-cli.
>>
>> This much should be pretty easy, since it's all ActiveRecord objects.
>> The only tricky part is if we store jobs in delayed_job as blobs of some
>> form, we might want to parse them out and show something user-friendly.
>>
>>> I also see us adding the ability to
>>> start/stop the queue and to adjust the number of workers through the
>>> command line.
>>
>> This sounds like it's more controlling the workers than the queue
>> itself, though I don't disagree with the objective at all.
>>
>>> Viewing the queue from Conductor would be nice, but
>>> that feels like follow-on work, especially if we foresee further
>>> structural changes to the UI.
>>
>> As with the rake task, this should really be pretty easy to implement,
>> though it doesn't seem like we have any equivalent pages right now.
>>
>>> To plug the recurring job hole in delayed_job, bringing in a
>>> scheduler like rufus-scheduler to queue jobs is a good idea.
>>
>> I will disclaim that I haven't worked with rufus-scheduler in the past,
>> but it looks like it's the type of thing we're looking for.
>>
>
> It seems that rufus-scheduler is based only on threads - it creates 
> one master thread which is only while-true loop with a sleep delay and 
> creation of threads for doing jobs. It's not a standalone process so 
> we would have to write a daemon anyway or run it from our rails 
> process. Overall my impression is that it would be step back in 
> compare to current dbomatic which at least uses separate processes.
We wouldn't use rufus-scheduler or another scheduler to run jobs. We 
would use the scheduler to queue jobs into delayed_job. The jobs would 
then then be picked up by workers in delayed_job. The number of 
delayed_job works is configurable and each worker runs in its own 
process and has its own rails environment.

The scheduling would need to be smart enough to not queue duplicate jobs 
in the queue if one is already queued or is running. That is something 
we would add.

A benefit I see with moving to jobs from dbomatic is it gives us an 
opportunity to break up the different pieces of functionality in 
dbomatic which will make it easier to test. It also gives us the 
opportunity to plug in alerts. Right now unless you look at the 
dbomatic.log it is hard know if a problem occurred.
>
> Delayed job doesn't support recurring jobs so if we decide for Delayed 
> job now, what will be next step in Dbomatic improvements? Try to use 
> Delayed job for recurring jobs too (by re-enqueing jobs)? Or Integrate 
> another background tool for running recurring jobs?
>

The thinking is combining a scheduler with delayed_job will get us 
recurring jobs.

> Using two background tools (one for delayed jobs and one for recurring 
> jobs) doesn't sound right to me - delayed and recurring jobs are very 
> similar to each other.
Using two background tools is not the goal.
>
> Using Delayed job for recurring jobs is not correct too. As mentioned 
> in a another thread, re-enqueing jobs is hacky and might be unsafe. 
> Though this option sounds still better than having two separate 
> background tools. I'm little bit afraid that logic which will have to 
> be written around recurring jobs will not be better than current 
> Dbomatic, but I could be wrong.

With resque, you don't get recurring jobs unless you add the 
resque-scheduler, a separate gem. So I think our approach here is 
similar to what you would get with resque.

I don't think re-queueing jobs is necessarily a bad idea, people have 
implemented it. It is a matter of trade-offs. But I think having a 
scheduler would be less work from a maintenance perspective

Until delayed_jobs supports recurring jobs natively, I think using the 
scheduler + delayed_job approach would be fine.
>
> Because of above I was most attracted by resque-scheduler or anything 
> else which supports both delayed and recurring jobs natively. As I 
> understand it, major problem with resque-scheduler is that Redis is 
> not in RHEL and it would be a problem to get it in, and also because 
> this tool is too heavyweight? And there is not another similar tool?
>
Yes, that is the general conclusion.

I don't oppose using resque, it just feels like there would be more road 
blocks to make it happen. Whereas we can run with delayed_jobs on day one.

>>> We'll
>>> need to think about how to handle alerts with recurring jobs. With
>>> dbomatic, errors are buried in the log. It is good that they are
>>> logged, but a person needs to dig around to figure out that there is
>>> a problem. If we add alerts, the recurring job would produce a stream
>>> of alerts/emails each time it failed, possibly flooding a person's
>>> inbox. We could add some logic to skip an alert if the same error
>>> occurred during the last run.
>>
>> Having worn a sysadmin hat in the past, I think a tidal wave of emails
>> to indicate a failure is really pretty common. (And it certainly gets
>> your attention more quickly than a single email about a potential
>> problem... Up until the mailserver crashes.)
>>
>> I think an easy work-around might be to only send alerts if the
>> type of task and the class of the exception hadn't previously been
>> emailed in the past N hours. The downside to this is that it isn't
>> immediately obvious if was a single occurrence of a freak failure, or if
>> it happened 2,500 times in the past hour.
>>
>> Suppose we created an Alerts table, and logged everything there? We
>> could use it for deciding if we should send an email or not, but it
>> would also allow an admin to log in and see "47,500 new notifications"
>> and see them? The header designs from the UX team showed a notification
>> counter, but we commented it out since we don't currently have an
>> analogous concept in Aeolus. This could implement it.
>>
>>> Speaking of alerts, is there a way to flag an alert with Conductor. I
>>> see there is a "Alerts" box in the Monitor overview page. How does
>>> that work exactly?
>>
>> Right now, I'm pretty sure that "Alerts" just lists Instance.failed.all
>> or something along those lines. It seems to me that we could implement
>> an Alert class, and we could (a) log it to the database, and (b) have an
>> after_create hook that decided if we should send an email or not. Does
>> that sound like it would work?
>>
>> -- Matt
>




More information about the aeolus-devel mailing list