On Jul 15, 2013, at 12:34 PM, Jay Shaughnessy <jshaughn(a)redhat.com> wrote:
On 7/15/2013 10:19 AM, Stefan Negrea wrote:
> Hello Everybody,
>
> I would need feedback for the percentage of disk space used metric for RHQ Storage
Nodes. The metric already went through three iterations and I think we could refine it
more before releasing next RHQ version.
>
>
> Background
> - The metric is Disk Used Percentage
> - It is a calculated metric
> - Data files are Cassandra's largest disk space consumers
> - Data files could be stored in one or more data directories, this is configurable by
the user
> - If multiple directories are configured, Cassandra will select the partition with
most disk space available
> - The size of all data files is available via the JMX interface
> - Disk space available is a system/platform property but to report anything for
Cassandra the focus needs to be on the partitions where data files are stored
>
> Iteration 1 - delivered with RHQ 4.8
> - The metric name: Calculated.DiskSpaceUsedPercentage
> - Calculation:
> + based on the overall disk usage, not just Cassandra (total disk used/total disk
available)
> + if multiple directories used for data files, just return max percentage
> - Problems:
> + the metric can be misleading, since it is reported in Cassandra sub-resource, yet
it represents a platform metric
> + the selection of max is not representative on how Cassandra works, since it
distributes data files across all the partitions available
> + using max will work correctly only in cases where there is only one data files
location
>
> Iteration 2 - Post RHQ 4.8
> - The metric name: Calculated.PartitionDiskSpaceUsedPercentage
> - Calculation:
> + an aggregate percentage of disk space used for all the partitions where Cassandra
stores data files
> + similar to iteration 1, but it looks at the disk space used across all the
partitions
> + example: for data files on two partitions: metric value = (disk space used of
partition_1 + partition_2)/(total disk space of partition_1 + partition_2)
> - Problems:
> + like Iteration 1, the metric can be misleading since it is a platform metric but
reported on Cassandra sub-resource
> + seeing a high percentage for disk utilization when you just installed a new RHQ
Storage Node could be confusing - "Cassandra is using 42% of the disk already?"
> + the only way for users to understand what is going on is to look at the metric
description, just looking at the table of values reported is not sufficient
> - Positives:
> + representative to how Cassandra uses disk partitions
> + an alert on this metric is guaranteed to trigger every time the free disk space
gets low
>
> Iteration 3 - in Master
> - The metric name: Calculated.PartitionDiskSpaceUsedPercentage
> - Calculation:
> + an aggregate percentage of disk space used by Cassandra data files for all the
partitions where Cassandra stores data files
> + similar to iteration 2, but it will not look at the overall disk space used, just
at the disk space used by data files
> + eg. for data files on two partitions: metric value = (disk space used by data
files)/(total disk space of partition_1 + partition_2)
> - Problems:
> + it is hard to design an alert for this metric because there is no percentage to
alert on that will guarantee trigger
> + for example: if total disk space utilization pre-Cassandra is 60%, a value of
50% of the alert on this metric will never be trigger the alert because there is only 40%
available disk
> + Cassandra data could grow until disk is full and never trigger an alert
> - Positives:
> + representative to how Cassandra uses disk partitions
> + the value will no longer be misleading, when a new RHQ Storage Node is installed
the disk used will low (eg. under 1%)
> + the user will not have to dig in documentation or report that the metric is wrong
out of the box
>
>
> Right now Iteration 3 is in master but I would need feedback since there is no
perfect solution (both Iteration 2 and Iteration 3 have downsides). I like the correctness
of the calculation for Iteration 3 but the alerting capability of Iteration 2. I do not
see any easy solution to get both...
>
>
> 1) Should we include both metrics? Overall disk space use & data files disk space
used? We could only hope that having the two metrics will make it easier for users to
understand what they are.
I think two metrics is a good idea. We must have a way to easily alert on a disk space
consumption. The Iteration-3 Calculated.PartitionDiskSpaceUsedPercentage metric is useful
information: How much of my used disk space is due to my storage node? But it's
mainly useful to help understand if a disk space issue is or is not due to the storage
node. We need the Iteration-3 Calculated.PartitionDiskSpaceUsedPercentage metric for
alerting on excessive disk space for the storage node partitions. Of course we'd need
two names that describe the distinction. I think both metrics should be on the same
resource as they are related.
I agree that we need two metrics. I do not think that they should be on the same resource
though because they are not necessarily related. It may not be the storage node that is
eating up space on the file system. The Calculated.PartitionDiskSpaceUsedPercentage metric
should go on the File System resource type.