discussion on measurement baseline calculation & problem metrics

Greg Hinkle ghinkle at redhat.com
Mon Aug 2 20:04:09 UTC 2010


I agree that data is almost always cyclical and most often patterned on time of day and day of week. I just meant that given our current functionality and level of granularity, a basic best practice would be a 7 day cycle calculated every 24 hours with a 24 hour delay. I think that's the best chance to keep useful data in the oob system in most cases... short of building something that can analyze for TOD, DOW and other cycles. A fancy system would be nice, but we can cheaply improve what we have for common cases.


-Greg


On Aug 2, 2010, at 3:20 PM, Joseph Marques wrote:

> >From #rhq earlier today...
> 
> (13:09:28) ghinkle: should our baseline system leave a gap on the running baselines? e.g. calculate a 7 day baseline based on 8 days ago to 1 day ago instead of 7 days to now
> (13:10:14) ghinkle: right now, all the oobs will essentially disappear after a baseline calc right?
> (13:35:08) mazz: we don't calc ALL baselines in one shot anymore
> (13:35:42) mazz: only those resources due to get their baselines to be recalced will get them done - and those OOBs would, I guess, go away for those resources
> (13:39:50) ghinkle: right... but if we did a delay we could still see things in the past 24 hours that are different from the historical baseline
> (13:40:05) ghinkle: i think the "delay gap" should be equal to recalculation period
> (13:40:38) ghinkle: and default it to recalcing every 24 hours for a 7 day period that ends 24 hours ago
> 
> -----
> 
> So if users want to calc baselines only every 3 days, would it then use X amount of data ending 3 days ago?
> 
> Keep in mind, there's a natural delay gap that comes into play here due to the fact that baselines are calculated using compressed data from the _1hr table, which only gets calculated at the top of the hour when the metric compression/purge job runs.
> 
> The major issues with baselines as I see them are as follows:
> 
> 1) baselines are a trailing average of data, but the fastest we can calculate them is once a day.  this is far too granular, it would be nice if we could calculate the trailing average over the last hour for every metric in the system, and then use that to compute whether incoming metrics are out-of-bounds / problems
> 
> 2) we don't distinguish between weekdays and weekends, nor do we distinguish between time of day.  for some services, it might make more sense to only use data from their known "active periods" to calculate the trailing average, which might only be during biz hours or excluding weekends.  so, on top of small trailing windows (1) it would also be nice to have separate blackout periods computed for the individual resources.
> 
> [I know that Heiko has had some thoughts about using a more sophisticated algorithm for this sliding window calculation.  I'm sure he'll chime in on this thread.]
> 
> 3) trailing averages don't make sense for all resources.  granted, we only calculate baselines for the DYNAMIC types (metrics that strictly strictly trend up or trend down are not candidates for baseline calculation), but it would be nice to give the user even finer control over this.  it should be possible to craft an interface that allows users to turn off baselines for resources altogether.  if not on the resource-by-resource level, perhaps allowing it to be toggled by individual resource type instead.
> 
> What do people think of these ideas?  Please respond with comments as well as if you have your own ideas for measurement baselines and/or problem metrics.
> 
> -joseph
> 
> For further reading on the topic, I've included what I felt were the most relevant links related to the original topic.  Keep in mind that baselines are a distinct concept (and data structure) from problem metrics (a.k.a out-of-bounds metrics).  However, as you'll see, measurement baseline data is one of the inputs to the algorithm that calculates problem metrics, so they are indeed related.
> 
> How baseline calculations work today...
> 
> http://www.rhq-project.org/display/JOPR2/FAQ#FAQ-WhendoBaselinesautocalculate%3F
> 
> What were some of the problems with the previous OOB / problem metrics subsystem...
> 
> http://www.rhq-project.org/display/RHQ/Design-BaselineOOBProblem
> 
> How does the current OOB / problem metrics subsystem function...
> 
> http://www.rhq-project.org/display/RHQ/Design-New+OOB
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel

- Greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://fedorahosted.org/pipermail/rhq-devel/attachments/20100802/cb0bf090/attachment.html 


More information about the rhq-devel mailing list