Alerts don't fire when REST is used to post data collections composed of multiple metric types to RHQ - UPDATED

Van Dillon vandillon at gmail.com
Wed Dec 17 20:34:17 UTC 2014


Hi,

I originally posted this question on 11/26 and have not received a
response.  I'm assuming my post was overlooked due to closeness to the
Thanksgiving holiday and the fact that you guys were really busy with RHQ
4.13 and RHQ.next.

I also have some additional information.  I've confirmed that the problem
occurs on RHQ 4.13.

If you'd prefer not to handle this problem here, I'd be happy to submit it
as a bug report.

Thanks,


Van Dillon

>>
>>
This is a follow up to a question I posted on 9/12 that involved using a
plugin that depends on the No-op plugin along with REST to push metrics to
RHQ.  The answer to that question was that resources have to be created as
children of an agent backed platform for alerts to work.

I'm running into a different problem with alerts that involves using REST
to post data collections to RHQ.  In summary, alerts don't fire when they
should when the data collection is composed of data of multiple metric
types.  I've tried both of the following methods on RHQ 4.10.0 and RHQ
4.12.0:

  POST /metric/data/raw
  POST /metric/data/raw/{resourceId}

Both these methods almost immediately call the same code in
MetricHandlerBean on the server side, so I'll just use test results for the
schedule id based method in my examples.

Let's say I have the following metric types and alert definitions:

 Metric: CPU Util (percent)  Alert: CPU Util > 50%
 Metric: Disk Reads Bytes    Alert: none

Here are the results from a test series of randomly generated data posted
at one minute intervals:

cpu util:0.49,  disk read bytes:700.0
cpu util:0.87,  disk read bytes:1180.0
cpu util:0.72,  disk read bytes:1210.0
cpu util:0.38,  disk read bytes:969.0
cpu util:0.10,  disk read bytes:806.0
+cpu util:0.52, disk read bytes:456.0
cpu util:0.09,  disk read bytes:696.0
cpu util:0.75,  disk read bytes:950.0
+cpu util:0.74, disk read bytes:297.0
cpu util:0.13,  disk read bytes:555.0

The '+' indicates data points that triggered alerts.  For the data points:
0.87, 0.72, and 0.75 no alert was triggered.

Another interesting thing happens if you add an alert for the "Disk Read
Bytes" metric: "Disk Read Bytes > 500".  In this case I posted the same
test series (composed of data points that all exceed the alert threshold
for both metrics) at one minute intervals:

cpu util:0.65,  +disk read bytes:700.0
cpu util:0.65,  +disk read bytes:700.0
cpu util:0.65,  +disk read bytes:700.0
cpu util:0.65,  +disk read bytes:700.0
+cpu util:0.65,  disk read bytes:700.0
cpu util:0.65,  +disk read bytes:700.0
+cpu util:0.65,  disk read bytes:700.0
cpu util:0.65,  +disk read bytes:700.0
cpu util:0.65,  +disk read bytes:700.0
+cpu util:0.65,  disk read bytes:700.0

Again, the '+' indicates data points that triggered alerts.  As you can
see, each post causes one alert to trigger.  I wanted to see if this
pattern holds up when more types of metrics are added, and indeed it does.
I added another metric called "Disk Read Ops" to the test series with an
alert defined as: "Disk Read Ops > 1000" and posted the following data at
one minute intervals:

cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
+cpu util:0.75,  disk read bytes:800.0,  disk read ops:1200.0
cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
cpu util:0.75,  +disk read bytes:800.0,  disk read ops:1200.0
+cpu util:0.75,  disk read bytes:800.0,  disk read ops:1200.0
cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
cpu util:0.75,  +disk read bytes:800.0,  disk read ops:1200.0


The only workaround I've been able to find for this problem is to only send
a single metric type per post when using the REST methods for raw data
collections.  When I do this, alerts always trigger when they should.
<<
<<
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20141217/31fe272f/attachment.html>


More information about the rhq-devel mailing list