On 04/12/2012 10:41 PM, Matt Wagner wrote:
On Thu, Apr 12, 2012 at 01:01:33PM -0700, Richard Su wrote:
>
> Thanks for the feedback Matt. Let's plan on going with delayed_jobs,
> unless someone dissents.
>
> We'll need to build some additional infrastructure around
> delayed_jobs to make it more usable. First would be a way to view the
> contents of the queues from the command line, maybe in the form of a
> rake task or through aeolus-cli.
This much should be pretty easy, since it's all ActiveRecord objects.
The only tricky part is if we store jobs in delayed_job as blobs of some
form, we might want to parse them out and show something user-friendly.
> I also see us adding the ability to
> start/stop the queue and to adjust the number of workers through the
> command line.
This sounds like it's more controlling the workers than the queue
itself, though I don't disagree with the objective at all.
> Viewing the queue from Conductor would be nice, but
> that feels like follow-on work, especially if we foresee further
> structural changes to the UI.
As with the rake task, this should really be pretty easy to implement,
though it doesn't seem like we have any equivalent pages right now.
> To plug the recurring job hole in delayed_job, bringing in a
> scheduler like rufus-scheduler to queue jobs is a good idea.
I will disclaim that I haven't worked with rufus-scheduler in the past,
but it looks like it's the type of thing we're looking for.
It seems that rufus-scheduler is based only on threads - it creates one
master thread which is only while-true loop with a sleep delay and
creation of threads for doing jobs. It's not a standalone process so we
would have to write a daemon anyway or run it from our rails process.
Overall my impression is that it would be step back in compare to
current dbomatic which at least uses separate processes.
Delayed job doesn't support recurring jobs so if we decide for Delayed
job now, what will be next step in Dbomatic improvements? Try to use
Delayed job for recurring jobs too (by re-enqueing jobs)? Or Integrate
another background tool for running recurring jobs?
Using two background tools (one for delayed jobs and one for recurring
jobs) doesn't sound right to me - delayed and recurring jobs are very
similar to each other.
Using Delayed job for recurring jobs is not correct too. As mentioned in
a another thread, re-enqueing jobs is hacky and might be unsafe. Though
this option sounds still better than having two separate background
tools. I'm little bit afraid that logic which will have to be written
around recurring jobs will not be better than current Dbomatic, but I
could be wrong.
Because of above I was most attracted by resque-scheduler or anything
else which supports both delayed and recurring jobs natively. As I
understand it, major problem with resque-scheduler is that Redis is not
in RHEL and it would be a problem to get it in, and also because this
tool is too heavyweight? And there is not another similar tool?
> We'll
> need to think about how to handle alerts with recurring jobs. With
> dbomatic, errors are buried in the log. It is good that they are
> logged, but a person needs to dig around to figure out that there is
> a problem. If we add alerts, the recurring job would produce a stream
> of alerts/emails each time it failed, possibly flooding a person's
> inbox. We could add some logic to skip an alert if the same error
> occurred during the last run.
Having worn a sysadmin hat in the past, I think a tidal wave of emails
to indicate a failure is really pretty common. (And it certainly gets
your attention more quickly than a single email about a potential
problem... Up until the mailserver crashes.)
I think an easy work-around might be to only send alerts if the
type of task and the class of the exception hadn't previously been
emailed in the past N hours. The downside to this is that it isn't
immediately obvious if was a single occurrence of a freak failure, or if
it happened 2,500 times in the past hour.
Suppose we created an Alerts table, and logged everything there? We
could use it for deciding if we should send an email or not, but it
would also allow an admin to log in and see "47,500 new notifications"
and see them? The header designs from the UX team showed a notification
counter, but we commented it out since we don't currently have an
analogous concept in Aeolus. This could implement it.
> Speaking of alerts, is there a way to flag an alert with Conductor. I
> see there is a "Alerts" box in the Monitor overview page. How does
> that work exactly?
Right now, I'm pretty sure that "Alerts" just lists Instance.failed.all
or something along those lines. It seems to me that we could implement
an Alert class, and we could (a) log it to the database, and (b) have an
after_create hook that decided if we should send an email or not. Does
that sound like it would work?
-- Matt