Storage Node - Disk Space Metric

Stefan Negrea snegrea at redhat.com
Tue Jul 16 13:54:00 UTC 2013



----- Original Message -----
> From: "John Sanda" <jsanda at redhat.com>
> To: rhq-devel at lists.fedorahosted.org
> Sent: Monday, July 15, 2013 4:41:57 PM
> Subject: Re: Storage Node - Disk Space Metric
> 
> 
> On Jul 15, 2013, at 5:03 PM, Stefan Negrea <snegrea at redhat.com> wrote:
> 
> > 
> > 
> > ----- Original Message -----
> >> From: "John Sanda" <jsanda at redhat.com>
> >> To: rhq-devel at lists.fedorahosted.org
> >> Sent: Monday, July 15, 2013 11:43:32 AM
> >> Subject: Re: Storage Node - Disk Space Metric
> >> 
> >>>> 
> >>>> 1) Should we include both metrics? Overall disk space use & data files
> >>>> disk space used? We could only hope that having the two metrics will
> >>>> make
> >>>> it easier for users to understand what they are.
> >>> 
> >>> I think two metrics is a good idea.  We must have a way to easily alert
> >>> on
> >>> a disk space consumption.  The Iteration-3
> >>> Calculated.PartitionDiskSpaceUsedPercentage metric is useful information:
> >>> How much of my used disk space is due to my storage node?   But it's
> >>> mainly useful to help understand if a disk space issue is or is not due
> >>> to
> >>> the storage node.  We need the Iteration-3
> >>> Calculated.PartitionDiskSpaceUsedPercentage metric for alerting on
> >>> excessive disk space for the storage node partitions. Of course we'd need
> >>> two names that describe the distinction.  I think both metrics should be
> >>> on the same resource as they are related.
> >> 
> >> 
> >> I agree that we need two metrics. I do not think that they should be on
> >> the
> >> same resource though because they are not necessarily related. It may not
> >> be
> >> the storage node that is eating up space on the file system. The
> >> Calculated.PartitionDiskSpaceUsedPercentage metric should go on the File
> >> System resource type.
> > 
> > 
> > I am not sure we can put any of these metrics outside of Storage Node
> > resources because they depend on the data files locations. A *nix
> > installation might have multiple mount points; but we would be interested
> > in alerting only for the mount points that have data files. To create an
> > alert we need to know the mount points for all data files, which is an RHQ
> > Storage Node property only. Data file locations can change over time and
> > we would need to be able to adjust this metric and alert on fly as the RHQ
> > Storage Node configuration evolves.
> > 
> > Adding alerts to all the partitions is not a good idea either because a
> > mount point like /boot could have a high usage percentage but never
> > contain Cassandra data files. Also, we will not be able to capture alerts
> > in the way we do it today (all alerts from the all the sub-resources of
> > RHQ Storage Nodes).
> > 
> > 
> > John, any thoughts on how to address these issues?
> > 
> 
> There are essentially two scenarios we want to cover. The first is surpassing
> some threshold due to growth of Cassandra data. If we set that threshold to
> 50% and Cassandra is using 257 MB of a 512 MB partition, then we should fire
> an alert. This alert definition belongs on the storage node type (or one of
> its descendants). The second scenario involves surpassing the threshold due
> to something other than Cassandra data. Cassandra might only be using 10 MB
> of a 512 MB, but since 257 MB overall is used we still want to fire an
> alert. I think this alert definition belongs on the File System resource
> type.
> 
> I think we handle multiple data directories and file systems no differently
> that a single data directory and a single file system. For each data
> directory, we determine the file system on which it is located and create an
> alert definition on that file system. If that data directory is removed,
> then we can disable the corresponding alert definition on the file system. I
> would not delete it in order to preserve history.

I am not sure how we could implement this actually. Today alerts are added to Storage Node resources and sub-resources on discovery. These alerts are created unconditionally without looking at the configuration; the creation is based on the resource type. I am not sure how we could create, update, and delete alerts that are on completely different resource trees (eg. platform) using complex logic. The complex logic would be to look at the data files location property on the Storage Service and make decisions about alerts on resources for another plugin.

> 
> With respect to where the alerts are defined, I do not see it as a problem
> that we have a pre-defined alert on a resource outside of the RHQ storage
> node type hierarchy. My thinking is this. Put the alert definition where it
> makes the most sense. One alert definition makes sense on the storage node
> because we are firing an alert due to the growth of Cassandra data. The
> other alert definition makes more sense on the File System type because we
> are not necessarily firing an alert due to Cassandra data. And it worth
> noting that we still want to fire the alert even if it is *not* due to
> Cassandra data. Cassandra could still run out of space; however, deploying
> another node to spread the load won't necessarily be an effective solution
> in this case.

Again, I am not sure about the implementation details on this. The main problem would be finding only alerts specific to Storage Nodes. Today is pretty simple, look at all the Storage Node resources. If we start having alerts all over the inventory, the task of deciding what is Storage Node alert and what is not will be become very difficult.


> 
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list