Storage Node - Disk Space Metric
Jay Shaughnessy
jshaughn at redhat.com
Mon Jul 15 16:34:51 UTC 2013
On 7/15/2013 10:19 AM, Stefan Negrea wrote:
> Hello Everybody,
>
> I would need feedback for the percentage of disk space used metric for RHQ Storage Nodes. The metric already went through three iterations and I think we could refine it more before releasing next RHQ version.
>
>
> Background
> - The metric is Disk Used Percentage
> - It is a calculated metric
> - Data files are Cassandra's largest disk space consumers
> - Data files could be stored in one or more data directories, this is configurable by the user
> - If multiple directories are configured, Cassandra will select the partition with most disk space available
> - The size of all data files is available via the JMX interface
> - Disk space available is a system/platform property but to report anything for Cassandra the focus needs to be on the partitions where data files are stored
>
> Iteration 1 - delivered with RHQ 4.8
> - The metric name: Calculated.DiskSpaceUsedPercentage
> - Calculation:
> + based on the overall disk usage, not just Cassandra (total disk used/total disk available)
> + if multiple directories used for data files, just return max percentage
> - Problems:
> + the metric can be misleading, since it is reported in Cassandra sub-resource, yet it represents a platform metric
> + the selection of max is not representative on how Cassandra works, since it distributes data files across all the partitions available
> + using max will work correctly only in cases where there is only one data files location
>
> Iteration 2 - Post RHQ 4.8
> - The metric name: Calculated.PartitionDiskSpaceUsedPercentage
> - Calculation:
> + an aggregate percentage of disk space used for all the partitions where Cassandra stores data files
> + similar to iteration 1, but it looks at the disk space used across all the partitions
> + example: for data files on two partitions: metric value = (disk space used of partition_1 + partition_2)/(total disk space of partition_1 + partition_2)
> - Problems:
> + like Iteration 1, the metric can be misleading since it is a platform metric but reported on Cassandra sub-resource
> + seeing a high percentage for disk utilization when you just installed a new RHQ Storage Node could be confusing - "Cassandra is using 42% of the disk already?"
> + the only way for users to understand what is going on is to look at the metric description, just looking at the table of values reported is not sufficient
> - Positives:
> + representative to how Cassandra uses disk partitions
> + an alert on this metric is guaranteed to trigger every time the free disk space gets low
>
> Iteration 3 - in Master
> - The metric name: Calculated.PartitionDiskSpaceUsedPercentage
> - Calculation:
> + an aggregate percentage of disk space used by Cassandra data files for all the partitions where Cassandra stores data files
> + similar to iteration 2, but it will not look at the overall disk space used, just at the disk space used by data files
> + eg. for data files on two partitions: metric value = (disk space used by data files)/(total disk space of partition_1 + partition_2)
> - Problems:
> + it is hard to design an alert for this metric because there is no percentage to alert on that will guarantee trigger
> + for example: if total disk space utilization pre-Cassandra is 60%, a value of 50% of the alert on this metric will never be trigger the alert because there is only 40% available disk
> + Cassandra data could grow until disk is full and never trigger an alert
> - Positives:
> + representative to how Cassandra uses disk partitions
> + the value will no longer be misleading, when a new RHQ Storage Node is installed the disk used will low (eg. under 1%)
> + the user will not have to dig in documentation or report that the metric is wrong out of the box
>
>
> Right now Iteration 3 is in master but I would need feedback since there is no perfect solution (both Iteration 2 and Iteration 3 have downsides). I like the correctness of the calculation for Iteration 3 but the alerting capability of Iteration 2. I do not see any easy solution to get both...
>
>
> 1) Should we include both metrics? Overall disk space use & data files disk space used? We could only hope that having the two metrics will make it easier for users to understand what they are.
I think two metrics is a good idea. We must have a way to easily alert
on a disk space consumption. The Iteration-3
Calculated.PartitionDiskSpaceUsedPercentage metric is useful
information: How much of my used disk space is due to my storage node?
But it's mainly useful to help understand if a disk space issue is or is
not due to the storage node. We need the Iteration-3
Calculated.PartitionDiskSpaceUsedPercentage metric for alerting on
excessive disk space for the storage node partitions. Of course we'd
need two names that describe the distinction. I think both metrics
should be on the same resource as they are related.
>
> 2) Is iteration 2 good? The confusion concern is not warranted?
>
> 3) Could we place the calculation done for Iteration 2 on a sub-resource that would not guarantee confusion?
>
> 4) Any other solution to have a self-documenting metric and a good alert template?
>
>
>
> Thank you,
> Stefan Negrea
>
> Software Engineer
>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
More information about the rhq-devel
mailing list