Storage Node - Disk Space Metric

John Sanda jsanda at redhat.com
Tue Jul 16 18:40:17 UTC 2013


On Jul 16, 2013, at 9:54 AM, Stefan Negrea <snegrea at redhat.com> wrote:

> 
> 
> ----- Original Message -----
>> From: "John Sanda" <jsanda at redhat.com>
>> To: rhq-devel at lists.fedorahosted.org
>> Sent: Monday, July 15, 2013 4:41:57 PM
>> Subject: Re: Storage Node - Disk Space Metric
>> 
>> 
>> On Jul 15, 2013, at 5:03 PM, Stefan Negrea <snegrea at redhat.com> wrote:
>> 
>>> 
>>> 
>>> ----- Original Message -----
>>>> From: "John Sanda" <jsanda at redhat.com>
>>>> To: rhq-devel at lists.fedorahosted.org
>>>> Sent: Monday, July 15, 2013 11:43:32 AM
>>>> Subject: Re: Storage Node - Disk Space Metric
>>>> 
>>>>>> 
>>>>>> 1) Should we include both metrics? Overall disk space use & data files
>>>>>> disk space used? We could only hope that having the two metrics will
>>>>>> make
>>>>>> it easier for users to understand what they are.
>>>>> 
>>>>> I think two metrics is a good idea.  We must have a way to easily alert
>>>>> on
>>>>> a disk space consumption.  The Iteration-3
>>>>> Calculated.PartitionDiskSpaceUsedPercentage metric is useful information:
>>>>> How much of my used disk space is due to my storage node?   But it's
>>>>> mainly useful to help understand if a disk space issue is or is not due
>>>>> to
>>>>> the storage node.  We need the Iteration-3
>>>>> Calculated.PartitionDiskSpaceUsedPercentage metric for alerting on
>>>>> excessive disk space for the storage node partitions. Of course we'd need
>>>>> two names that describe the distinction.  I think both metrics should be
>>>>> on the same resource as they are related.
>>>> 
>>>> 
>>>> I agree that we need two metrics. I do not think that they should be on
>>>> the
>>>> same resource though because they are not necessarily related. It may not
>>>> be
>>>> the storage node that is eating up space on the file system. The
>>>> Calculated.PartitionDiskSpaceUsedPercentage metric should go on the File
>>>> System resource type.
>>> 
>>> 
>>> I am not sure we can put any of these metrics outside of Storage Node
>>> resources because they depend on the data files locations. A *nix
>>> installation might have multiple mount points; but we would be interested
>>> in alerting only for the mount points that have data files. To create an
>>> alert we need to know the mount points for all data files, which is an RHQ
>>> Storage Node property only. Data file locations can change over time and
>>> we would need to be able to adjust this metric and alert on fly as the RHQ
>>> Storage Node configuration evolves.
>>> 
>>> Adding alerts to all the partitions is not a good idea either because a
>>> mount point like /boot could have a high usage percentage but never
>>> contain Cassandra data files. Also, we will not be able to capture alerts
>>> in the way we do it today (all alerts from the all the sub-resources of
>>> RHQ Storage Nodes).
>>> 
>>> 
>>> John, any thoughts on how to address these issues?
>>> 
>> 
>> There are essentially two scenarios we want to cover. The first is surpassing
>> some threshold due to growth of Cassandra data. If we set that threshold to
>> 50% and Cassandra is using 257 MB of a 512 MB partition, then we should fire
>> an alert. This alert definition belongs on the storage node type (or one of
>> its descendants). The second scenario involves surpassing the threshold due
>> to something other than Cassandra data. Cassandra might only be using 10 MB
>> of a 512 MB, but since 257 MB overall is used we still want to fire an
>> alert. I think this alert definition belongs on the File System resource
>> type.
>> 
>> I think we handle multiple data directories and file systems no differently
>> that a single data directory and a single file system. For each data
>> directory, we determine the file system on which it is located and create an
>> alert definition on that file system. If that data directory is removed,
>> then we can disable the corresponding alert definition on the file system. I
>> would not delete it in order to preserve history.
> 
> I am not sure how we could implement this actually. Today alerts are added to Storage Node resources and sub-resources on discovery. These alerts are created unconditionally without looking at the configuration; the creation is based on the resource type. I am not sure how we could create, update, and delete alerts that are on completely different resource trees (eg. platform) using complex logic. The complex logic would be to look at the data files location property on the Storage Service and make decisions about alerts on resources for another plugin.

I spent some time investigating this. I think that the difficulty in the correlation comes from the storage node and file system getting reported to the server in different inventory reports. When the agent starts up, it kicks off an initial server scan, in which the storage node should be discovered. The agent subsequently kicks off an initial service scan, in which the file system where data files are stored should get reported to the server. I think it is entirely possible for the File System resource to be committed into inventory before the Storage Node and vice-versa. This makes doing the correlation a lot more complex, and I think simplicity of putting the metrics and alert definition on the storage node resource outweigh the complexity of putting the free space metric/alert definition on the File System resource. In the event we get negative feedback from users on this, we can always revisit at some point in the future.

I also had some concern about the overall free space metric including all partitions where Cassandra data is stored as opposed to having a free space metric per file system. I have confirmed that when data files are generated both as a result of a table being flushed to disk and as a result of compaction, Cassandra uses the same algorithm in both cases. It writes the new files to the partition with the most free space. The free space metrics across multiple partitions works out well with this. 

> 
>> 
>> With respect to where the alerts are defined, I do not see it as a problem
>> that we have a pre-defined alert on a resource outside of the RHQ storage
>> node type hierarchy. My thinking is this. Put the alert definition where it
>> makes the most sense. One alert definition makes sense on the storage node
>> because we are firing an alert due to the growth of Cassandra data. The
>> other alert definition makes more sense on the File System type because we
>> are not necessarily firing an alert due to Cassandra data. And it worth
>> noting that we still want to fire the alert even if it is *not* due to
>> Cassandra data. Cassandra could still run out of space; however, deploying
>> another node to spread the load won't necessarily be an effective solution
>> in this case.
> 
> Again, I am not sure about the implementation details on this. The main problem would be finding only alerts specific to Storage Nodes. Today is pretty simple, look at all the Storage Node resources. If we start having alerts all over the inventory, the task of deciding what is Storage Node alert and what is not will be become very difficult.
> 
> 
>> 
>> 
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
>> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel



More information about the rhq-devel mailing list