long term scalability thoughts

John Sanda jsanda at redhat.com
Wed Aug 10 21:22:49 UTC 2011


Several thoughts....

First, that quote strikes me as misinformed. The "old school" database 
be it oracle or (preferably) postgresql is perfectly capable of handling 
large amounts of data with great performance. With that said, there are 
obviously situations as with metrics where performance *is* an issue. 
And with metrics the database will eventually become the bottleneck.

Secondly, as for other solutions like some of the ones mentioned, you 
can very easily wind up with lousy performance, maybe even more so than 
with a RDBMS. Some of these not so old school technologies are designed 
to address specific problem spaces and are not nearly as general purpose 
as a RDBMS. One of the draws to these other technologies is that they 
are designed to scale out horizontally much more easily than you can do 
with a RDBMS. Much of that ease of scaling out stems from the fact these 
technologies often favor availability over consistency whereas the RDBMS 
favors consistency. Backing off of consistency requirements makes it a 
lot easier to build a scalable, distributed system.

Third, if people want to manage an environment twice their current size, 
I think it is an oversimplification to simply say add more machines. It 
is not an easy problem to solve, and in fact if we already have a good 
solution here, I am aware of it. Then again, this is a really big 
project and there are plenty of new things I am continually discovering :-)

Lastly, I want to point out the approach we are taking with the drift 
subsystem. Persistence of drift entities and content is not managed 
directly through our SLSB layer but instead through server plugins. The 
default plugin does use the SLSB layer and RHQ database. This might be 
suitable for most users. I have also been working on another server 
plugin that uses MongoDB as its persistence store. This might be an 
attractive alternative if/when scalability becomes an issue. MongoDB 
provides a nice solution in terms of scalability using sharding coupled 
with master-slave replication. The key thing to take away though is that 
a flexible infrastructure is being put in place that allows users to 
plug in different back ends with different performance and scalability 
characteristics.

- John


On 8/10/11 4:17 PM, Charles Crouch wrote:
> A paragraph from an unrelated email this morning helped solidify my thinking around where we could potentially end up going wrt to performance and scalability of RHQ...
>
> "MongoDB, Membase, Memcache allow data flexibility and cloud scale. These are not old-sk00l Oracle databases that get slow with lots of data. These scale horizontally and are enabled by the scaling of a PaaS. We do of course have MySQL for the bits of data that aren't going to go through the roof."
>
> If users want to manage 1m metrics per minute, or 1m events per minute or 1m alerts per minute they are going to need N machines across which to scale the load. But at the same time there are probably going to be areas of the data model that just will never need to scale like that e.g. plugins, potentially the entire inventory (the requirements for reading/writing 1m resources could be very different to 30m *new* metrics every 12hours) and for that data something like a Postgres instance maybe sufficient. As long as we can also scale down the whole architecture to a single box for those people with smaller environments and still have a reliable and performant system.
>
> BTW I'm not talking about making RHQ into a SaaS here or anything PaaS related right now. Just how to build a system that users could install on their own hardware that would scale in a close to linear fashion. e.g. If people want to manage an environment twice their current size, they call up Dell and order N more machines, set them up and install RHQ.
>
> One further point, I think even if N can get quite large, I think users would still see that as a reasonable trade-off for near linear scalability (assuming that's achievable). If people have a really big environment they are more likely to invest in a larger monitoring/management infrastructure.
>
> More thoughts please...
>
> Thanks
> Charles
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel



More information about the rhq-devel mailing list