Storage Node - Disk Space Metric

John Sanda jsanda at redhat.com
Mon Jul 15 21:41:57 UTC 2013


On Jul 15, 2013, at 5:03 PM, Stefan Negrea <snegrea at redhat.com> wrote:

> 
> 
> ----- Original Message -----
>> From: "John Sanda" <jsanda at redhat.com>
>> To: rhq-devel at lists.fedorahosted.org
>> Sent: Monday, July 15, 2013 11:43:32 AM
>> Subject: Re: Storage Node - Disk Space Metric
>> 
>>>> 
>>>> 1) Should we include both metrics? Overall disk space use & data files
>>>> disk space used? We could only hope that having the two metrics will make
>>>> it easier for users to understand what they are.
>>> 
>>> I think two metrics is a good idea.  We must have a way to easily alert on
>>> a disk space consumption.  The Iteration-3
>>> Calculated.PartitionDiskSpaceUsedPercentage metric is useful information:
>>> How much of my used disk space is due to my storage node?   But it's
>>> mainly useful to help understand if a disk space issue is or is not due to
>>> the storage node.  We need the Iteration-3
>>> Calculated.PartitionDiskSpaceUsedPercentage metric for alerting on
>>> excessive disk space for the storage node partitions. Of course we'd need
>>> two names that describe the distinction.  I think both metrics should be
>>> on the same resource as they are related.
>> 
>> 
>> I agree that we need two metrics. I do not think that they should be on the
>> same resource though because they are not necessarily related. It may not be
>> the storage node that is eating up space on the file system. The
>> Calculated.PartitionDiskSpaceUsedPercentage metric should go on the File
>> System resource type.
> 
> 
> I am not sure we can put any of these metrics outside of Storage Node resources because they depend on the data files locations. A *nix installation might have multiple mount points; but we would be interested in alerting only for the mount points that have data files. To create an alert we need to know the mount points for all data files, which is an RHQ Storage Node property only. Data file locations can change over time and we would need to be able to adjust this metric and alert on fly as the RHQ Storage Node configuration evolves. 
> 
> Adding alerts to all the partitions is not a good idea either because a mount point like /boot could have a high usage percentage but never contain Cassandra data files. Also, we will not be able to capture alerts in the way we do it today (all alerts from the all the sub-resources of RHQ Storage Nodes).
> 
> 
> John, any thoughts on how to address these issues?
> 

There are essentially two scenarios we want to cover. The first is surpassing some threshold due to growth of Cassandra data. If we set that threshold to 50% and Cassandra is using 257 MB of a 512 MB partition, then we should fire an alert. This alert definition belongs on the storage node type (or one of its descendants). The second scenario involves surpassing the threshold due to something other than Cassandra data. Cassandra might only be using 10 MB of a 512 MB, but since 257 MB overall is used we still want to fire an alert. I think this alert definition belongs on the File System resource type.

I think we handle multiple data directories and file systems no differently that a single data directory and a single file system. For each data directory, we determine the file system on which it is located and create an alert definition on that file system. If that data directory is removed, then we can disable the corresponding alert definition on the file system. I would not delete it in order to preserve history.

With respect to where the alerts are defined, I do not see it as a problem that we have a pre-defined alert on a resource outside of the RHQ storage node type hierarchy. My thinking is this. Put the alert definition where it makes the most sense. One alert definition makes sense on the storage node because we are firing an alert due to the growth of Cassandra data. The other alert definition makes more sense on the File System type because we are not necessarily firing an alert due to Cassandra data. And it worth noting that we still want to fire the alert even if it is *not* due to Cassandra data. Cassandra could still run out of space; however, deploying another node to spread the load won't necessarily be an effective solution in this case.




More information about the rhq-devel mailing list