discussion on measurement baseline calculation & problem metrics

Joseph Marques jmarques at redhat.com
Mon Aug 2 19:20:39 UTC 2010


 From #rhq earlier today...

(13:09:28) ghinkle: should our baseline system leave a gap on the 
running baselines? e.g. calculate a 7 day baseline based on 8 days ago 
to 1 day ago instead of 7 days to now
(13:10:14) ghinkle: right now, all the oobs will essentially disappear 
after a baseline calc right?
(13:35:08) mazz: we don't calc ALL baselines in one shot anymore
(13:35:42) mazz: only those resources due to get their baselines to be 
recalced will get them done - and those OOBs would, I guess, go away for 
those resources
(13:39:50) ghinkle: right... but if we did a delay we could still see 
things in the past 24 hours that are different from the historical baseline
(13:40:05) ghinkle: i think the "delay gap" should be equal to 
recalculation period
(13:40:38) ghinkle: and default it to recalcing every 24 hours for a 7 
day period that ends 24 hours ago

-----

So if users want to calc baselines only every 3 days, would it then use 
X amount of data ending 3 days ago?

Keep in mind, there's a natural delay gap that comes into play here due 
to the fact that baselines are calculated using compressed data from the 
_1hr table, which only gets calculated at the top of the hour when the 
metric compression/purge job runs.

The major issues with baselines as I see them are as follows:

1) baselines are a trailing average of data, but the fastest we can 
calculate them is once a day.  this is far too granular, it would be 
nice if we could calculate the trailing average over the last hour for 
every metric in the system, and then use that to compute whether 
incoming metrics are out-of-bounds / problems

2) we don't distinguish between weekdays and weekends, nor do we 
distinguish between time of day.  for some services, it might make more 
sense to only use data from their known "active periods" to calculate 
the trailing average, which might only be during biz hours or excluding 
weekends.  so, on top of small trailing windows (1) it would also be 
nice to have separate blackout periods computed for the individual 
resources.

[I know that Heiko has had some thoughts about using a more 
sophisticated algorithm for this sliding window calculation.  I'm sure 
he'll chime in on this thread.]

3) trailing averages don't make sense for all resources.  granted, we 
only calculate baselines for the DYNAMIC types (metrics that strictly 
strictly trend up or trend down are not candidates for baseline 
calculation), but it would be nice to give the user even finer control 
over this.  it should be possible to craft an interface that allows 
users to turn off baselines for resources altogether.  if not on the 
resource-by-resource level, perhaps allowing it to be toggled by 
individual resource type instead.

What do people think of these ideas?  Please respond with comments as 
well as if you have your own ideas for measurement baselines and/or 
problem metrics.

-joseph

For further reading on the topic, I've included what I felt were the 
most relevant links related to the original topic.  Keep in mind that 
baselines are a distinct concept (and data structure) from problem 
metrics (a.k.a out-of-bounds metrics).  However, as you'll see, 
measurement baseline data is one of the inputs to the algorithm that 
calculates problem metrics, so they are indeed related.

How baseline calculations work today...

http://www.rhq-project.org/display/JOPR2/FAQ#FAQ-WhendoBaselinesautocalculate%3F

What were some of the problems with the previous OOB / problem metrics 
subsystem...

http://www.rhq-project.org/display/RHQ/Design-BaselineOOBProblem

How does the current OOB / problem metrics subsystem function...

http://www.rhq-project.org/display/RHQ/Design-New+OOB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://fedorahosted.org/pipermail/rhq-devel/attachments/20100802/8c04cbbc/attachment.html 


More information about the rhq-devel mailing list