Metrics Migration Tool - Cassandra

Stefan Negrea snegrea at redhat.com
Mon Jan 14 22:58:11 UTC 2013



----- Original Message -----
> From: "Alan Santos" <asantos at redhat.com>
> To: rhq-devel at lists.fedorahosted.org
> Sent: Monday, January 14, 2013 1:27:41 PM
> Subject: Re: Metrics Migration Tool - Cassandra
> 
> You Jay Shaughnessy, I like the way you think.
> 
> That sounds like minimal downtime (the usual) which is not related to
> the size of database but with a *temporary* delay in historic
> measurements.
> 
> This is why I had asked whether it would be possible to keep both
> metric storage options, basically to make the transition in the
> background.  This sounds much less complicated.
> 
> -alan
> 
> 
> ----- Original Message -----
> From: "Jay Shaughnessy" <jshaughn at redhat.com>
> To: rhq-devel at lists.fedorahosted.org
> Sent: Monday, January 14, 2013 1:33:49 PM
> Subject: Re: Metrics Migration Tool - Cassandra
> 
> 
> Stefan, has any thought been given to (optionally) migrating the bulk
> of
> the data *after* the upgrade?   I'm not sure if this make sense,
> really.  But the idea being that the customer upgrades RHQ and starts
> running with the new backend.  At this point no data is being written
> to
> the legacy tables so they are quiet.  All new metric data will move
> to
> the new Cassandra backend.  Then, with RHQ running we migrate the
> legacy
> data into the new backend.  So, for some period we lack historical
> data,
> but it would hopefully not be a very large delay.  It's possible
> maybe
> that until the migration is complete we suspend aggregation.  Just a
> thought.

If RHQ server is up and running normally things get complicated. Because historical data would not be not migrated before start, any aggregation that runs until the data is migrated for a specific resource will be skewed to only include new data collected. Since the rows from the old database are to be migrated in no specific order, there is no way to predict how many aggregates will get skewed in this process. And the aggregate computation will cascade. Lets say that two 1-hours aggregates use limited data, that will translate into at least one 6-hour aggregate being skewed, and at least one 24-hour aggregate skewed. The reason why I use the word skewed and not wrong is because the aggregates would not be wrong, they would just be based on a limited amount of data. Furthermore, this also impacts baseline computations since they are based on 1-hour aggregates. Also OOB alerting will be impacted since the baselines will be calculated based on limited data. So false alerts could be triggered or no alert at all be triggered due to baselines with small data samples.

Now we considered running RHQ but with the aggregate service down. In this case RHQ could continue to collect raw metrics. But that creates other problems. Since baselines are based on 1h hour aggregates, baseline computations will have to turned of too. Since baselines will not be computed until the migration is over then OOB alerts will not be fired until all the data is migrated.


On the other hand, if the RHQ server is down then metrics will still be collected at agent level and the server will get them after the upgrade. The advantage here is that migration process will be faster since the relation database, Cassandra, and system resources (disk and CPU) will be dedicated almost exclusively to the migration.


So, is it better to stop aggregation, baseline computations, and have no OOB alerts until the migration is over knowing that the migration process might run longer because RHQ is running? Or it is better to run a faster migration with RHQ server down? 


> 
> 
> On 1/14/2013 12:00 PM, Elias Ross wrote:
> > Just my 2 cents:
> >
> > I would like to continue running RHQ with minimal downtime (less
> > than
> > 30 minutes if possible). The main issue with downtime is loss of
> > systems monitoring capability. Historical data is definitely nice
> > to
> > have, but system monitoring trumps this. I would sooner drop the
> > metrics data than have to face an unpredictable migration process.
> >
> > I would prefer if migration could happen concurrent to running of
> > RHQ.
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list