On Thu, Apr 12, 2012 at 01:01:33PM -0700, Richard Su wrote:
Thanks for the feedback Matt. Let's plan on going with delayed_jobs,
unless someone dissents.
We'll need to build some additional infrastructure around
delayed_jobs to make it more usable. First would be a way to view the
contents of the queues from the command line, maybe in the form of a
rake task or through aeolus-cli.
This much should be pretty easy, since it's all ActiveRecord objects.
The only tricky part is if we store jobs in delayed_job as blobs of some
form, we might want to parse them out and show something user-friendly.
I also see us adding the ability to
start/stop the queue and to adjust the number of workers through the
command line.
This sounds like it's more controlling the workers than the queue
itself, though I don't disagree with the objective at all.
Viewing the queue from Conductor would be nice, but
that feels like follow-on work, especially if we foresee further
structural changes to the UI.
As with the rake task, this should really be pretty easy to implement,
though it doesn't seem like we have any equivalent pages right now.
To plug the recurring job hole in delayed_job, bringing in a
scheduler like rufus-scheduler to queue jobs is a good idea.
I will disclaim that I haven't worked with rufus-scheduler in the past,
but it looks like it's the type of thing we're looking for.
We'll
need to think about how to handle alerts with recurring jobs. With
dbomatic, errors are buried in the log. It is good that they are
logged, but a person needs to dig around to figure out that there is
a problem. If we add alerts, the recurring job would produce a stream
of alerts/emails each time it failed, possibly flooding a person's
inbox. We could add some logic to skip an alert if the same error
occurred during the last run.
Having worn a sysadmin hat in the past, I think a tidal wave of emails
to indicate a failure is really pretty common. (And it certainly gets
your attention more quickly than a single email about a potential
problem... Up until the mailserver crashes.)
I think an easy work-around might be to only send alerts if the
type of task and the class of the exception hadn't previously been
emailed in the past N hours. The downside to this is that it isn't
immediately obvious if was a single occurrence of a freak failure, or if
it happened 2,500 times in the past hour.
Suppose we created an Alerts table, and logged everything there? We
could use it for deciding if we should send an email or not, but it
would also allow an admin to log in and see "47,500 new notifications"
and see them? The header designs from the UX team showed a notification
counter, but we commented it out since we don't currently have an
analogous concept in Aeolus. This could implement it.
Speaking of alerts, is there a way to flag an alert with Conductor.
I
see there is a "Alerts" box in the Monitor overview page. How does
that work exactly?
Right now, I'm pretty sure that "Alerts" just lists Instance.failed.all
or something along those lines. It seems to me that we could implement
an Alert class, and we could (a) log it to the database, and (b) have an
after_create hook that decided if we should send an email or not. Does
that sound like it would work?
-- Matt