Scalability issues in RHQ 4.9 - A summary

John Sanda jsanda at redhat.com
Sat Mar 8 02:45:06 UTC 2014


On Mar 7, 2014, at 8:59 PM, Elias Ross <genman at noderunner.net> wrote:

> I've encountered the following scalability issues:
> 
> Bug 1025918 Uninventoring resources is slow; should not take more than a second
> 
> This was fixed by improving the query. Basically only is an issue when
> number of resources is in the 100,000+ range.
> 
> Bug 1064563 - Separate metrics compression and OOB from actual data purge tasks
> 
> This is something fairly simple to do. These jobs don't need to be run
> hourly. They also don't belong with the metrics compression process.
> The deletes don't scale well, and there's also no way to turn them off
> completely.

I could see separating compression from baseline and OOB calculations; however, things are coupled together. Baselines could be calculated independently of the compression job, but OOB calculations depends on both baselines and the one hour metrics computed during compression.
> 
> Bug 1073093 - Admin -> Metrics/Alert/Drift query does not scale well
> when alert definition count high
> 
> Problem happens with a large number of alert definitions. Fixed by
> adding a couple of indexes. Somebody needs to add indexes to
> Hibernate/upgrade script.
> 
> Bug 1073558 - Purge process for deleting alert definitions can lead to
> high database contention and timeouts
> 
> Not addressed. I'm thinking there are a couple of options here:
> 1) Purge a few definitions at a time, not every one.
> 2) Figure out if some indexes need to be added.
> 3) Drop some of the foreign key constraints. Not really needed and
> hurts overall performance. My DBAs say that "This schema was designed
> using the old way."
> 
> Bug 1070473 - When updating resource-type based alerts, ORA-00060:
> deadlock detected when deleting rhq_config
> 
> It's possible in the UI to have multiple 'save' clicks cause two
> transactions to run in contention. This can really hit hard when you
> happen to be doing this during a purge.
> 
> No bugs for these:
> 1) Event Purging is slow. Hopefully addressed by moving this to
> Cassandra release. Addressed by adding indexes. Can be caused by too
> many rows in rhq_event_source, and foreign
> 2) Trait Purging is slow, and can cause database transactions to halt.
> Meaning, the agents can't report metrics, and the server can run out
> of database connections. The work around is to stop this from
> happening. Again, should be addressed with moving this stuff to
> Cassandra.
> 3) Aggregation slow (can run > 1 hour for enough metrics.) Being
> addressed well enough in 4.11, I assume.
> 4) OOB slow. There was Bug 1059412 but doesn't necessarily scale that
> well. OOB moving to Cassandra, though, may help.

Events, traits, baselines, and OOBs are on the list of things to migrate to Cassandra. Are you still seeing it take over an hour with the performance changes in master that will go into 4.10? The schema changes targeted for 4.11 should have a big impact as they will dramatically reduce the number of reads performed during aggregation.

> 
> ...
> 
> I'd like to help with 1073558 (purge alert definitions), but not sure
> the recommended approach. My suggestion would be to only purge ~1000
> or so per transaction.
> 
> I wouldn't mind working on some Cassandra features either. I could
> work on moving events to Cassandra, but I'm not sure how interested I
> am in dealing with the data migration steps or whatnot.

I welcome/encourage you to start some design discussion about migrating events. Don't let the data migration portion discourage you. It will be a big effort, and I am sure someone else would gladly jump in to work on that part.

> 
> Anyway, any advice would be welcome here.
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel



More information about the rhq-devel mailing list