metrics database discussion

Elias Ross genman at noderunner.net
Mon Jun 11 19:02:29 UTC 2012


On Mon, Jun 11, 2012 at 8:46 AM, John Sanda <jsanda at redhat.com> wrote:

> The measurement subsystem presents challenges in terms of performance.
> Persisting metric data is write-intensive. The database eventually becomes
> a bottleneck as we try to store more and more metric data. Optimizations
> have been made over time to address performance problems, but at the end of
> the day, the database still remains the bottleneck.  Below is a bullet
> point list of things we need to consider and discuss as we start the
> process of looking for the next generation metrics database.
>

If you want to store everything in ONE database I can't help you, sorry.
But for metrics storage, I don't see why transaction support is required,
and dev tools and modeling seem less important as well. I would ignore
those 'requirements'.


Grid storage platforms like HBase (the Hadoop Database) would be perfect to
use for storing metrics, events, and the like that don't require
transactions and don't require join queries.

Unfortunately installing and maintaining an HBase cluster is a large
undertaking, similar in complexity and difficultly to an Oracle cluster.

So something like HBase would be a big win WRT:

* Performance
* Scalability

Not so much for:

* Development Tools
* Data Modelling and Querying
* Management

One nice thing about Map-Reduce also is you can effectively offload
computations (summary data, computing averages and variance) to the cluster
and not involve the client (RHQ Server in this case.)

I would ship out-of-box with Infinispan and provide support for HBase and
probably some other data grid technology. The use of Infinispan makes the
most sense, as that project would benefit as well and RedHat gets to own
the entire stack.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://fedorahosted.org/pipermail/rhq-devel/attachments/20120611/5575c7f0/attachment.html>


More information about the rhq-devel mailing list