----- Original Message -----
From: "Larry O'Leary" <loleary(a)redhat.com>
To: rhq-devel(a)lists.fedorahosted.org
Sent: Tuesday, July 16, 2013 6:22:06 PM
Subject: Re: Storage Node - Disk Space Metric
On Mon, 2013-07-15 at 18:58 -0400, John Sanda wrote:
> On Jul 15, 2013, at 5:46 PM, Jiri Kremser <jkremser(a)redhat.com> wrote:
>
> > Basically I am +1 on having both metrics, both can be useful for setting
> > up custom alerts.
> >
> > But I am concerned that the percentage based metric is not proper for
> > alerting because of its relative nature. While it make sense when
> > calculating the heap size based alerts, it is different in disk space,
> > because differences in partition sizes could be enormous.
> > "Calculated.SystemDiskSpaceUsedPercentage > 90%" may mean many
things on
> > different HW. I know that the alerts are adjustable and one could change
> > the number to whatever she wants, but we should provide smart defaults.
> >
> > What about having another one called for instance
> > "Calculated.SystemDiskSpaceAvailable" that would be calculated as
the
> > sum of free spaces across all partitions where data dirs for C* are
> > located? (because of the way how C* selects the data dir) And trigger
> > the alert if the number is lower than X, where X could be the number of
> > megabytes necessary for storing the data for, say, 7 days. Using this
> > approach, we could have alert notifications saying something like "If
> > you don't add more disks, you will run out of space in approx 1
week.".
>
> A metric that considers the sum of free space across all partitions would
> not address the increases in disk usage (albeit temporary) due to
> compaction.
I am curious to how this is different then looking at a percentage on
individual partitions? For example, when looking at disk usage for a
traditional relational database, a warning is not seen until either disk
space has run out or it has resulted in data being moved from one
partition to another. The hope is that it would be the latter and you
could then increase said partition's space or add new partitions. In the
end, the database doesn't keep track of how much space is there and how
much of the total space it has used up.
Perhaps we are trying to do too much? The expectation is that the user
provides enough disk space for the storage node to function. Hopefully
they will create an alert on their disk resources that will provide them
with this information. If so, perhaps a metric that provides the sum of
the free space across all the partitions used by the storage node is
sufficient?
Cassandra is a little bit different than Postgres. The data files can be stored in
multiple directories. Because data is the only one that grows significantly (compared to
caches for example) it is the only configuration option that accepts multiple directories.
If a user configures multiple data directories Cassandra selects the directory on the
partition with the most free space. Looking at the overall disk space is not enough, that
information needs to be used in conjunction with the current Cassandra configuration. For
this reason, we will need a targeted metric and alert; alerts at the platform level would
be too broad and sometimes irrelevant.
Running data compaction will take significantly more space: "during compaction, there
is a temporary spike in disk space usage and disk I/O."
(
http://www.datastax.com/docs/1.2/operations/compaction_compression). Even if there is
enough space to store data files and accommodate a moderate increase in data, there might
not be enough space to run compaction. So keeping track of the some sort of metric for the
amount free disk space left on the partitions where data files is stored is important.
I think the metrics and alerts align with our overall goal to black-box Cassandra. If we
provide all these out-of-box along with instructions on what to do, then users do not have
to learn the inner workings of Cassandra.
Thank you,
Stefan Negrea