Scalability issues in RHQ 4.9 - A summary

Elias Ross genman at noderunner.net
Sat Mar 8 01:59:58 UTC 2014


I've encountered the following scalability issues:

Bug 1025918 Uninventoring resources is slow; should not take more than a second

This was fixed by improving the query. Basically only is an issue when
number of resources is in the 100,000+ range.

Bug 1064563 - Separate metrics compression and OOB from actual data purge tasks

This is something fairly simple to do. These jobs don't need to be run
hourly. They also don't belong with the metrics compression process.
The deletes don't scale well, and there's also no way to turn them off
completely.

Bug 1073093 - Admin -> Metrics/Alert/Drift query does not scale well
when alert definition count high

Problem happens with a large number of alert definitions. Fixed by
adding a couple of indexes. Somebody needs to add indexes to
Hibernate/upgrade script.

Bug 1073558 - Purge process for deleting alert definitions can lead to
high database contention and timeouts

Not addressed. I'm thinking there are a couple of options here:
1) Purge a few definitions at a time, not every one.
2) Figure out if some indexes need to be added.
3) Drop some of the foreign key constraints. Not really needed and
hurts overall performance. My DBAs say that "This schema was designed
using the old way."

Bug 1070473 - When updating resource-type based alerts, ORA-00060:
deadlock detected when deleting rhq_config

It's possible in the UI to have multiple 'save' clicks cause two
transactions to run in contention. This can really hit hard when you
happen to be doing this during a purge.

No bugs for these:
1) Event Purging is slow. Hopefully addressed by moving this to
Cassandra release. Addressed by adding indexes. Can be caused by too
many rows in rhq_event_source, and foreign
2) Trait Purging is slow, and can cause database transactions to halt.
Meaning, the agents can't report metrics, and the server can run out
of database connections. The work around is to stop this from
happening. Again, should be addressed with moving this stuff to
Cassandra.
3) Aggregation slow (can run > 1 hour for enough metrics.) Being
addressed well enough in 4.11, I assume.
4) OOB slow. There was Bug 1059412 but doesn't necessarily scale that
well. OOB moving to Cassandra, though, may help.

...

I'd like to help with 1073558 (purge alert definitions), but not sure
the recommended approach. My suggestion would be to only purge ~1000
or so per transaction.

I wouldn't mind working on some Cassandra features either. I could
work on moving events to Cassandra, but I'm not sure how interested I
am in dealing with the data migration steps or whatnot.

Anyway, any advice would be welcome here.


More information about the rhq-devel mailing list