Alerts don't fire when REST is used to post data collections composed of multiple metric types to RHQ - UPDATED

Michael Burman miburman at redhat.com
Thu Dec 18 07:06:26 UTC 2014


Hi,

Please file a BZ as this doesn't look like intended behaviour. There has 
been some work in PR 51 to improve workflow for alerting for metrics 
coming from the REST interface, but it's not merged yet. It might solve 
your issue (if you have time to test that branch, then great).

   - MIcke

On 17.12.2014 22:34, Van Dillon wrote:
> Hi,
>
> I originally posted this question on 11/26 and have not received a 
> response.  I'm assuming my post was overlooked due to closeness to the 
> Thanksgiving holiday and the fact that you guys were really busy with 
> RHQ 4.13 and RHQ.next.
>
> I also have some additional information.  I've confirmed that the 
> problem occurs on RHQ 4.13.
>
> If you'd prefer not to handle this problem here, I'd be happy to 
> submit it as a bug report.
>
> Thanks,
>
>
> Van Dillon
>
> >>
> >>
> This is a follow up to a question I posted on 9/12 that involved using 
> a plugin that depends on the No-op plugin along with REST to push 
> metrics to RHQ.  The answer to that question was that resources have 
> to be created as children of an agent backed platform for alerts to work.
>
> I'm running into a different problem with alerts that involves using 
> REST to post data collections to RHQ.  In summary, alerts don't fire 
> when they should when the data collection is composed of data of 
> multiple metric types.  I've tried both of the following methods on 
> RHQ 4.10.0 and RHQ 4.12.0:
>
>   POST /metric/data/raw
>   POST /metric/data/raw/{resourceId}
>
> Both these methods almost immediately call the same code in 
> MetricHandlerBean on the server side, so I'll just use test results 
> for the schedule id based method in my examples.
>
> Let's say I have the following metric types and alert definitions:
>
>  Metric: CPU Util (percent)  Alert: CPU Util > 50%
>  Metric: Disk Reads Bytes    Alert: none
> Here are the results from a test series of randomly generated data 
> posted at one minute intervals:
>
> cpu util:0.49,  disk read bytes:700.0
> cpu util:0.87,  disk read bytes:1180.0
> cpu util:0.72,  disk read bytes:1210.0
> cpu util:0.38,  disk read bytes:969.0
> cpu util:0.10,  disk read bytes:806.0
> +cpu util:0.52, disk read bytes:456.0
> cpu util:0.09,  disk read bytes:696.0
> cpu util:0.75,  disk read bytes:950.0
> +cpu util:0.74, disk read bytes:297.0
> cpu util:0.13,  disk read bytes:555.0
> The '+' indicates data points that triggered alerts.  For the data 
> points: 0.87, 0.72, and 0.75 no alert was triggered.
>
> Another interesting thing happens if you add an alert for the "Disk 
> Read Bytes" metric: "Disk Read Bytes > 500".  In this case I posted 
> the same test series (composed of data points that all exceed the 
> alert threshold for both metrics) at one minute intervals:
>
> cpu util:0.65,  +disk read bytes:700.0
> cpu util:0.65,  +disk read bytes:700.0
> cpu util:0.65,  +disk read bytes:700.0
> cpu util:0.65,  +disk read bytes:700.0
> +cpu util:0.65,  disk read bytes:700.0
> cpu util:0.65,  +disk read bytes:700.0
> +cpu util:0.65,  disk read bytes:700.0
> cpu util:0.65,  +disk read bytes:700.0
> cpu util:0.65,  +disk read bytes:700.0
> +cpu util:0.65,  disk read bytes:700.0
>
> Again, the '+' indicates data points that triggered alerts.  As you 
> can see, each post causes one alert to trigger.  I wanted to see if 
> this pattern holds up when more types of metrics are added, and indeed 
> it does.  I added another metric called "Disk Read Ops" to the test 
> series with an alert defined as: "Disk Read Ops > 1000" and posted the 
> following data at one minute intervals:
>
> cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
> cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
> cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
> +cpu util:0.75,  disk read bytes:800.0,  disk read ops:1200.0
> cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
> cpu util:0.75,  +disk read bytes:800.0,  disk read ops:1200.0
> +cpu util:0.75,  disk read bytes:800.0,  disk read ops:1200.0
> cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
> cpu util:0.75,   disk read bytes:800.0, +disk read ops:1200.0
> cpu util:0.75,  +disk read bytes:800.0,  disk read ops:1200.0
>
>
> The only workaround I've been able to find for this problem is to only 
> send a single metric type per post when using the REST methods for raw 
> data collections.  When I do this, alerts always trigger when they should.
> <<
> <<
>
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20141218/5d200ad6/attachment.html>


More information about the rhq-devel mailing list