Storage Node - Disk Space Metric

Larry O'Leary loleary at redhat.com
Wed Jul 17 23:51:52 UTC 2013


On Wed, 2013-07-17 at 16:51 -0400, Stefan Negrea wrote:
> 
> ----- Original Message -----
> > From: "Larry O'Leary" <loleary at redhat.com>
> > To: rhq-devel at lists.fedorahosted.org
> > Sent: Tuesday, July 16, 2013 6:22:06 PM
> > Subject: Re: Storage Node - Disk Space Metric
> > 
> > On Mon, 2013-07-15 at 18:58 -0400, John Sanda wrote:
> > > On Jul 15, 2013, at 5:46 PM, Jiri Kremser <jkremser at redhat.com> wrote:
> > > 
> > > > Basically I am +1 on having both metrics, both can be useful for setting
> > > > up custom alerts.
> > > > 
> > > > But I am concerned that the percentage based metric is not proper for
> > > > alerting because of its relative nature. While it make sense when
> > > > calculating the heap size based alerts, it is different in disk space,
> > > > because differences in partition sizes could be enormous.
> > > > "Calculated.SystemDiskSpaceUsedPercentage > 90%" may mean many things on
> > > > different HW. I know that the alerts are adjustable and one could change
> > > > the number to whatever she wants, but we should provide smart defaults.
> > > > 
> > > > What about having another one called for instance
> > > > "Calculated.SystemDiskSpaceAvailable" that would be calculated as the
> > > > sum of free spaces across all partitions where data dirs for C* are
> > > > located? (because of the way how C* selects the data dir) And trigger
> > > > the alert if the number is lower than X, where X could be the number of
> > > > megabytes necessary for storing the data for, say, 7 days. Using this
> > > > approach, we could have alert notifications saying something like "If
> > > > you don't add more disks, you will run out of space in approx 1 week.".
> > > 
> > > A metric that considers the sum of free space across all partitions would
> > > not address the increases in disk usage (albeit temporary) due to
> > > compaction.
> > 
> > I am curious to how this is different then looking at a percentage on
> > individual partitions? For example, when looking at disk usage for a
> > traditional relational database, a warning is not seen until either disk
> > space has run out or it has resulted in data being moved from one
> > partition to another. The hope is that it would be the latter and you
> > could then increase said partition's space or add new partitions. In the
> > end, the database doesn't keep track of how much space is there and how
> > much of the total space it has used up.
> > 
> > Perhaps we are trying to do too much? The expectation is that the user
> > provides enough disk space for the storage node to function. Hopefully
> > they will create an alert on their disk resources that will provide them
> > with this information. If so, perhaps a metric that provides the sum of
> > the free space across all the partitions used by the storage node is
> > sufficient?
> > 
> > 
> 
> Cassandra is a little bit different than Postgres. The data files can
> be stored in multiple directories. Because data is the only one that
> grows significantly (compared to caches for example) it is the only
> configuration option that accepts multiple directories. If a user
> configures multiple data directories Cassandra selects the directory
> on the partition with the most free space. Looking at the overall disk
> space is not enough, that information needs to be used in conjunction
> with the current Cassandra configuration. For this reason, we will
> need a targeted metric and alert; alerts at the platform level would
> be too broad and sometimes irrelevant. 

Actually, you can span PostgreSQL data files across
directories/partitions. But I wasn't tying to compare what Cassandra
needs to do to what a RDBMS does. Only that the user needs to ensure
that there is enough space for the data store to do its work.

> Running data compaction will take significantly more space: "during
> compaction, there is a temporary spike in disk space usage and disk
> I/O." (http://www.datastax.com/docs/1.2/operations/compaction_compression). Even if there is enough space to store data files and accommodate a moderate increase in data, there might not be enough space to run compaction. So keeping track of the some sort of metric for the amount free disk space left on the partitions where data files is stored is important.  
> 
> 
> I think the metrics and alerts align with our overall goal to
> black-box Cassandra. If we provide all these out-of-box along with
> instructions on what to do, then users do not have to learn the inner
> workings of Cassandra.

>From the sound of it, what is really needed is a way to know when the
data store is in jeopardy of exceeding the available space on a single
partition. Essentially a way to compare one metric against another along
with a threshold calculation.

This could be doable assuming that you know how much space would be
needed to perform compaction and compression in a worst case scenario.
Are we talking about the data store potentially doubling in size as a
worse case scenario? Couldn't the alert then just be based on:

(<total data store size> * 2.2) > <largest available space in a
directory> = trigger alert

This assumes the data could grow to 2 times its size and would need to
reside in a single file during compaction with a 20% overhead as a
threshold -- configurable by the user of course but with a sane
default. 

Then we could get away from reporting metrics on available space
altogether. 

-- 
Larry O'Leary
https://plus.google.com/u/0/112645929986009801513



More information about the rhq-devel mailing list