Aggregate alerting feature - Bug 1019472

Elias Ross genman at noderunner.net
Thu Dec 5 18:49:50 UTC 2013


Hi, I know you've seen a lot of patches from me. I've been doing some major
work on getting our fairly large RHQ 4.9 installation working.

Speaking of fairly large, one feature that RHQ lacks is alerting based on
group average metrics. I added a server plugin feature that can alert on
this data.

One obvious use case: You have a group of 10 hosts serving web traffic. If
the amount of traffic drops during a 10 minute period from the previous
day, say by 30%, you would like to be alerted. Of course during the day
itself there are traffic fluctuations by more than 30%, so comparisons
against an absolute baseline isn't particularly useful. It also isn't
useful to alert per host, as some hosts may be receiving no traffic or
increased traffic, depending on load balance configurations, server
upgrades, or whatever. Or having to manage 10 alerts (or more) isn't easy.

The way the plugin works (and you can see the documentation here:
https://bugzilla.redhat.com/show_bug.cgi?id=1019472 ) is every 5 minutes,
there are calculations made using the most recent data, and yesterday's
data. (There are other rules, such as comparing last week's data, or
absolute value comparisons, or availability checks like the percentage of
up servers.) Then when an alert occurs, an alert definition is created at
the resource group level (since RHQ has no UI support for this), and an
alert is sent.

What I have is fully functional, but required some patches to RHQ core to
support alert notifications for resource groups.

I don't know if this feature is in demand, but it would be nice to see the
following introduced, in order:
0) The feature included as part of the server plugin suite, but perhaps
disabled by default.
1) Patches included to fix alerting for resource groups. (Patches are part
of the bug.)
2) UI support for listing and clearing alerts for resource groups. (Partial
patches in the bug, should be easy to fix.)
3) Alert definitions created through the UI, not through resource tags.
This requires expanding the types of alert definitions RHQ formally
supports.
4) An API for adding alert definitions through the command line interface.
(Because adding a tag to a resource group is supported through the CLI.)

Thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20131205/dcfed880/attachment.html>


More information about the rhq-devel mailing list