Storage Node - Disk Space Metric

John Sanda jsanda at redhat.com
Mon Jul 15 16:43:32 UTC 2013


On Jul 15, 2013, at 12:34 PM, Jay Shaughnessy <jshaughn at redhat.com> wrote:

> 
> On 7/15/2013 10:19 AM, Stefan Negrea wrote:
>> Hello Everybody,
>> 
>> I would need feedback for the percentage of disk space used metric for RHQ Storage Nodes. The metric already went through three iterations and I think we could refine it more before releasing next RHQ version.
>> 
>> 
>> Background
>> - The metric is Disk Used Percentage
>> - It is a calculated metric
>> - Data files are Cassandra's largest disk space consumers
>> - Data files could be stored in one or more data directories, this is configurable by the user
>> - If multiple directories are configured, Cassandra will select the partition with most disk space available
>> - The size of all data files is available via the JMX interface
>> - Disk space available is a system/platform property but to report anything for Cassandra the focus needs to be on the partitions where data files are stored
>> 
>> Iteration 1 - delivered with RHQ 4.8
>> - The metric name:   Calculated.DiskSpaceUsedPercentage
>> - Calculation:
>>   + based on the overall disk usage, not just Cassandra (total disk used/total disk available)
>>   + if multiple directories used for data files, just return max percentage
>> - Problems:
>>   + the metric can be misleading, since it is reported in Cassandra sub-resource, yet it represents a platform metric
>>   + the selection of max is not representative on how Cassandra works, since it distributes data files across all the partitions available
>>   + using max will work correctly only in cases where there is only one data files location
>> 
>> Iteration 2 - Post RHQ 4.8
>> - The metric name:  Calculated.PartitionDiskSpaceUsedPercentage
>> - Calculation:
>>   + an aggregate percentage of disk space used for all the partitions where Cassandra stores data files
>>   + similar to iteration 1, but it looks at the disk space used across all the partitions
>>   + example: for data files on two partitions:  metric value = (disk space used of partition_1 + partition_2)/(total disk space of partition_1 + partition_2)
>> - Problems:
>>   + like Iteration 1, the metric can be misleading since it is a platform metric but reported on Cassandra sub-resource
>>   + seeing a high percentage for disk utilization when you just installed a new RHQ Storage Node could be confusing - "Cassandra is using 42% of the disk already?"
>>   + the only way for users to understand what is going on is to look at the metric description, just looking at the table of values reported is not sufficient
>> - Positives:
>>   + representative to how Cassandra uses disk partitions
>>   + an alert on this metric is guaranteed to trigger every time the free disk space gets low
>> 
>> Iteration 3 - in Master
>> - The metric name:  Calculated.PartitionDiskSpaceUsedPercentage
>> - Calculation:
>>   + an aggregate percentage of disk space used by Cassandra data files for all the partitions where Cassandra stores data files
>>   + similar to iteration 2, but it will not look at the overall disk space used, just at the disk space used by data files
>>   + eg. for data files on two partitions: metric value = (disk space used by data files)/(total disk space of partition_1 + partition_2)
>> - Problems:
>>   + it is hard to design an alert for this metric because there is no percentage to alert on that will guarantee trigger
>>   + for example:  if total disk space utilization pre-Cassandra is 60%, a value of 50% of the alert on this metric will never be trigger the alert because there is only 40% available disk
>>   + Cassandra data could grow until disk is full and never trigger an alert
>> - Positives:
>>   + representative to how Cassandra uses disk partitions
>>   + the value will no longer be misleading, when a new RHQ Storage Node is installed the disk used will low (eg. under 1%)
>>   + the user will not have to dig in documentation or report that the metric is wrong out of the box
>> 
>> 
>> Right now Iteration 3 is in master but I would need feedback since there is no perfect solution (both Iteration 2 and Iteration 3 have downsides). I like the correctness of the calculation for Iteration 3 but the alerting capability of Iteration 2. I do not see any easy solution to get both...
>> 
>> 
>> 1) Should we include both metrics? Overall disk space use & data files disk space used? We could only hope that having the two metrics will make it easier for users to understand what they are.
> 
> I think two metrics is a good idea.  We must have a way to easily alert on a disk space consumption.  The Iteration-3 Calculated.PartitionDiskSpaceUsedPercentage metric is useful information: How much of my used disk space is due to my storage node?   But it's mainly useful to help understand if a disk space issue is or is not due to the storage node.  We need the Iteration-3 Calculated.PartitionDiskSpaceUsedPercentage metric for alerting on excessive disk space for the storage node partitions. Of course we'd need two names that describe the distinction.  I think both metrics should be on the same resource as they are related.


I agree that we need two metrics. I do not think that they should be on the same resource though because they are not necessarily related. It may not be the storage node that is eating up space on the file system. The Calculated.PartitionDiskSpaceUsedPercentage metric should go on the File System resource type.


More information about the rhq-devel mailing list