RFC: Background Processing

Jan Provaznik jprovazn at redhat.com
Mon Apr 16 07:28:39 UTC 2012


On 04/13/2012 10:22 PM, Richard Su wrote:
> On 04/13/2012 03:58 AM, Jan Provazník wrote:
>> On 04/12/2012 10:41 PM, Matt Wagner wrote:
>>> On Thu, Apr 12, 2012 at 01:01:33PM -0700, Richard Su wrote:
>>>>
>>>> Thanks for the feedback Matt. Let's plan on going with delayed_jobs,
>>>> unless someone dissents.
>>>>
>>>> We'll need to build some additional infrastructure around
>>>> delayed_jobs to make it more usable. First would be a way to view the
>>>> contents of the queues from the command line, maybe in the form of a
>>>> rake task or through aeolus-cli.
>>>
>>> This much should be pretty easy, since it's all ActiveRecord objects.
>>> The only tricky part is if we store jobs in delayed_job as blobs of some
>>> form, we might want to parse them out and show something user-friendly.
>>>
>>>> I also see us adding the ability to
>>>> start/stop the queue and to adjust the number of workers through the
>>>> command line.
>>>
>>> This sounds like it's more controlling the workers than the queue
>>> itself, though I don't disagree with the objective at all.
>>>
>>>> Viewing the queue from Conductor would be nice, but
>>>> that feels like follow-on work, especially if we foresee further
>>>> structural changes to the UI.
>>>
>>> As with the rake task, this should really be pretty easy to implement,
>>> though it doesn't seem like we have any equivalent pages right now.
>>>
>>>> To plug the recurring job hole in delayed_job, bringing in a
>>>> scheduler like rufus-scheduler to queue jobs is a good idea.
>>>
>>> I will disclaim that I haven't worked with rufus-scheduler in the past,
>>> but it looks like it's the type of thing we're looking for.
>>>
>>
>> It seems that rufus-scheduler is based only on threads - it creates
>> one master thread which is only while-true loop with a sleep delay and
>> creation of threads for doing jobs. It's not a standalone process so
>> we would have to write a daemon anyway or run it from our rails
>> process. Overall my impression is that it would be step back in
>> compare to current dbomatic which at least uses separate processes.
> We wouldn't use rufus-scheduler or another scheduler to run jobs. We
> would use the scheduler to queue jobs into delayed_job. The jobs would

OK, sounds like a plan.

> then then be picked up by workers in delayed_job. The number of
> delayed_job works is configurable and each worker runs in its own
> process and has its own rails environment.
>
> The scheduling would need to be smart enough to not queue duplicate jobs
> in the queue if one is already queued or is running. That is something
> we would add.
>
> A benefit I see with moving to jobs from dbomatic is it gives us an
> opportunity to break up the different pieces of functionality in
> dbomatic which will make it easier to test. It also gives us the
> opportunity to plug in alerts. Right now unless you look at the
> dbomatic.log it is hard know if a problem occurred.
>>
>> Delayed job doesn't support recurring jobs so if we decide for Delayed
>> job now, what will be next step in Dbomatic improvements? Try to use
>> Delayed job for recurring jobs too (by re-enqueing jobs)? Or Integrate
>> another background tool for running recurring jobs?
>>
>
> The thinking is combining a scheduler with delayed_job will get us
> recurring jobs.
>
>> Using two background tools (one for delayed jobs and one for recurring
>> jobs) doesn't sound right to me - delayed and recurring jobs are very
>> similar to each other.
> Using two background tools is not the goal.
>>
>> Using Delayed job for recurring jobs is not correct too. As mentioned
>> in a another thread, re-enqueing jobs is hacky and might be unsafe.
>> Though this option sounds still better than having two separate
>> background tools. I'm little bit afraid that logic which will have to
>> be written around recurring jobs will not be better than current
>> Dbomatic, but I could be wrong.
>
> With resque, you don't get recurring jobs unless you add the
> resque-scheduler, a separate gem. So I think our approach here is
> similar to what you would get with resque.
>
> I don't think re-queueing jobs is necessarily a bad idea, people have
> implemented it. It is a matter of trade-offs. But I think having a
> scheduler would be less work from a maintenance perspective
>
> Until delayed_jobs supports recurring jobs natively, I think using the
> scheduler + delayed_job approach would be fine.
>>
>> Because of above I was most attracted by resque-scheduler or anything
>> else which supports both delayed and recurring jobs natively. As I
>> understand it, major problem with resque-scheduler is that Redis is
>> not in RHEL and it would be a problem to get it in, and also because
>> this tool is too heavyweight? And there is not another similar tool?
>>
> Yes, that is the general conclusion.
>
> I don't oppose using resque, it just feels like there would be more road
> blocks to make it happen. Whereas we can run with delayed_jobs on day one.
>
>>>> We'll
>>>> need to think about how to handle alerts with recurring jobs. With
>>>> dbomatic, errors are buried in the log. It is good that they are
>>>> logged, but a person needs to dig around to figure out that there is
>>>> a problem. If we add alerts, the recurring job would produce a stream
>>>> of alerts/emails each time it failed, possibly flooding a person's
>>>> inbox. We could add some logic to skip an alert if the same error
>>>> occurred during the last run.
>>>
>>> Having worn a sysadmin hat in the past, I think a tidal wave of emails
>>> to indicate a failure is really pretty common. (And it certainly gets
>>> your attention more quickly than a single email about a potential
>>> problem... Up until the mailserver crashes.)
>>>
>>> I think an easy work-around might be to only send alerts if the
>>> type of task and the class of the exception hadn't previously been
>>> emailed in the past N hours. The downside to this is that it isn't
>>> immediately obvious if was a single occurrence of a freak failure, or if
>>> it happened 2,500 times in the past hour.
>>>
>>> Suppose we created an Alerts table, and logged everything there? We
>>> could use it for deciding if we should send an email or not, but it
>>> would also allow an admin to log in and see "47,500 new notifications"
>>> and see them? The header designs from the UX team showed a notification
>>> counter, but we commented it out since we don't currently have an
>>> analogous concept in Aeolus. This could implement it.
>>>
>>>> Speaking of alerts, is there a way to flag an alert with Conductor. I
>>>> see there is a "Alerts" box in the Monitor overview page. How does
>>>> that work exactly?
>>>
>>> Right now, I'm pretty sure that "Alerts" just lists Instance.failed.all
>>> or something along those lines. It seems to me that we could implement
>>> an Alert class, and we could (a) log it to the database, and (b) have an
>>> after_create hook that decided if we should send an email or not. Does
>>> that sound like it would work?
>>>
>>> -- Matt
>>
>




More information about the aeolus-devel mailing list