RFC: Background Processing

Imre Farkas ifarkas at redhat.com
Thu Jun 14 07:05:45 UTC 2012


On 04/16/2012 09:28 AM, Jan Provaznik wrote:
> On 04/13/2012 10:22 PM, Richard Su wrote:
>> On 04/13/2012 03:58 AM, Jan Provazník wrote:
>>> On 04/12/2012 10:41 PM, Matt Wagner wrote:
>>>> On Thu, Apr 12, 2012 at 01:01:33PM -0700, Richard Su wrote:
>>>>>
>>>>> Thanks for the feedback Matt. Let's plan on going with delayed_jobs,
>>>>> unless someone dissents.
>>>>>
>>>>> We'll need to build some additional infrastructure around
>>>>> delayed_jobs to make it more usable. First would be a way to view the
>>>>> contents of the queues from the command line, maybe in the form of a
>>>>> rake task or through aeolus-cli.
>>>>
>>>> This much should be pretty easy, since it's all ActiveRecord objects.
>>>> The only tricky part is if we store jobs in delayed_job as blobs of
>>>> some
>>>> form, we might want to parse them out and show something user-friendly.
>>>>
>>>>> I also see us adding the ability to
>>>>> start/stop the queue and to adjust the number of workers through the
>>>>> command line.
>>>>
>>>> This sounds like it's more controlling the workers than the queue
>>>> itself, though I don't disagree with the objective at all.
>>>>
>>>>> Viewing the queue from Conductor would be nice, but
>>>>> that feels like follow-on work, especially if we foresee further
>>>>> structural changes to the UI.
>>>>
>>>> As with the rake task, this should really be pretty easy to implement,
>>>> though it doesn't seem like we have any equivalent pages right now.
>>>>
>>>>> To plug the recurring job hole in delayed_job, bringing in a
>>>>> scheduler like rufus-scheduler to queue jobs is a good idea.
>>>>
>>>> I will disclaim that I haven't worked with rufus-scheduler in the past,
>>>> but it looks like it's the type of thing we're looking for.
>>>>
>>>
>>> It seems that rufus-scheduler is based only on threads - it creates
>>> one master thread which is only while-true loop with a sleep delay and
>>> creation of threads for doing jobs. It's not a standalone process so
>>> we would have to write a daemon anyway or run it from our rails
>>> process. Overall my impression is that it would be step back in
>>> compare to current dbomatic which at least uses separate processes.
>> We wouldn't use rufus-scheduler or another scheduler to run jobs. We
>> would use the scheduler to queue jobs into delayed_job. The jobs would
>
> OK, sounds like a plan.
>
>> then then be picked up by workers in delayed_job. The number of
>> delayed_job works is configurable and each worker runs in its own
>> process and has its own rails environment.
>>
>> The scheduling would need to be smart enough to not queue duplicate jobs
>> in the queue if one is already queued or is running. That is something
>> we would add.
>>
>> A benefit I see with moving to jobs from dbomatic is it gives us an
>> opportunity to break up the different pieces of functionality in
>> dbomatic which will make it easier to test. It also gives us the
>> opportunity to plug in alerts. Right now unless you look at the
>> dbomatic.log it is hard know if a problem occurred.
>>>
>>> Delayed job doesn't support recurring jobs so if we decide for Delayed
>>> job now, what will be next step in Dbomatic improvements? Try to use
>>> Delayed job for recurring jobs too (by re-enqueing jobs)? Or Integrate
>>> another background tool for running recurring jobs?
>>>
>>
>> The thinking is combining a scheduler with delayed_job will get us
>> recurring jobs.
>>
>>> Using two background tools (one for delayed jobs and one for recurring
>>> jobs) doesn't sound right to me - delayed and recurring jobs are very
>>> similar to each other.
>> Using two background tools is not the goal.
>>>
>>> Using Delayed job for recurring jobs is not correct too. As mentioned
>>> in a another thread, re-enqueing jobs is hacky and might be unsafe.
>>> Though this option sounds still better than having two separate
>>> background tools. I'm little bit afraid that logic which will have to
>>> be written around recurring jobs will not be better than current
>>> Dbomatic, but I could be wrong.
>>
>> With resque, you don't get recurring jobs unless you add the
>> resque-scheduler, a separate gem. So I think our approach here is
>> similar to what you would get with resque.
>>
>> I don't think re-queueing jobs is necessarily a bad idea, people have
>> implemented it. It is a matter of trade-offs. But I think having a
>> scheduler would be less work from a maintenance perspective
>>
>> Until delayed_jobs supports recurring jobs natively, I think using the
>> scheduler + delayed_job approach would be fine.
>>>
>>> Because of above I was most attracted by resque-scheduler or anything
>>> else which supports both delayed and recurring jobs natively. As I
>>> understand it, major problem with resque-scheduler is that Redis is
>>> not in RHEL and it would be a problem to get it in, and also because
>>> this tool is too heavyweight? And there is not another similar tool?
>>>
>> Yes, that is the general conclusion.
>>
>> I don't oppose using resque, it just feels like there would be more road
>> blocks to make it happen. Whereas we can run with delayed_jobs on day
>> one.
>>
>>>>> We'll
>>>>> need to think about how to handle alerts with recurring jobs. With
>>>>> dbomatic, errors are buried in the log. It is good that they are
>>>>> logged, but a person needs to dig around to figure out that there is
>>>>> a problem. If we add alerts, the recurring job would produce a stream
>>>>> of alerts/emails each time it failed, possibly flooding a person's
>>>>> inbox. We could add some logic to skip an alert if the same error
>>>>> occurred during the last run.
>>>>
>>>> Having worn a sysadmin hat in the past, I think a tidal wave of emails
>>>> to indicate a failure is really pretty common. (And it certainly gets
>>>> your attention more quickly than a single email about a potential
>>>> problem... Up until the mailserver crashes.)
>>>>
>>>> I think an easy work-around might be to only send alerts if the
>>>> type of task and the class of the exception hadn't previously been
>>>> emailed in the past N hours. The downside to this is that it isn't
>>>> immediately obvious if was a single occurrence of a freak failure,
>>>> or if
>>>> it happened 2,500 times in the past hour.
>>>>
>>>> Suppose we created an Alerts table, and logged everything there? We
>>>> could use it for deciding if we should send an email or not, but it
>>>> would also allow an admin to log in and see "47,500 new notifications"
>>>> and see them? The header designs from the UX team showed a notification
>>>> counter, but we commented it out since we don't currently have an
>>>> analogous concept in Aeolus. This could implement it.
>>>>
>>>>> Speaking of alerts, is there a way to flag an alert with Conductor. I
>>>>> see there is a "Alerts" box in the Monitor overview page. How does
>>>>> that work exactly?
>>>>
>>>> Right now, I'm pretty sure that "Alerts" just lists Instance.failed.all
>>>> or something along those lines. It seems to me that we could implement
>>>> an Alert class, and we could (a) log it to the database, and (b)
>>>> have an
>>>> after_create hook that decided if we should send an email or not. Does
>>>> that sound like it would work?
>>>>
>>>> -- Matt
>>>
>>
>

I think we should take into consideration the fact that in Rails 4 there 
is going to be some unified queuing API: 
https://github.com/rails/rails/commit/adff4a706a5d7ad18ef05303461e1a0d848bd662

Imre



More information about the aeolus-devel mailing list