----- Original Message -----
From: "Alan Santos" <asantos(a)redhat.com>
To: rhq-devel(a)lists.fedorahosted.org
Sent: Monday, January 14, 2013 1:27:41 PM
Subject: Re: Metrics Migration Tool - Cassandra
You Jay Shaughnessy, I like the way you think.
That sounds like minimal downtime (the usual) which is not related to
the size of database but with a *temporary* delay in historic
measurements.
This is why I had asked whether it would be possible to keep both
metric storage options, basically to make the transition in the
background. This sounds much less complicated.
-alan
----- Original Message -----
From: "Jay Shaughnessy" <jshaughn(a)redhat.com>
To: rhq-devel(a)lists.fedorahosted.org
Sent: Monday, January 14, 2013 1:33:49 PM
Subject: Re: Metrics Migration Tool - Cassandra
Stefan, has any thought been given to (optionally) migrating the bulk
of
the data *after* the upgrade? I'm not sure if this make sense,
really. But the idea being that the customer upgrades RHQ and starts
running with the new backend. At this point no data is being written
to
the legacy tables so they are quiet. All new metric data will move
to
the new Cassandra backend. Then, with RHQ running we migrate the
legacy
data into the new backend. So, for some period we lack historical
data,
but it would hopefully not be a very large delay. It's possible
maybe
that until the migration is complete we suspend aggregation. Just a
thought.
If RHQ server is up and running normally things get complicated. Because historical data
would not be not migrated before start, any aggregation that runs until the data is
migrated for a specific resource will be skewed to only include new data collected. Since
the rows from the old database are to be migrated in no specific order, there is no way to
predict how many aggregates will get skewed in this process. And the aggregate computation
will cascade. Lets say that two 1-hours aggregates use limited data, that will translate
into at least one 6-hour aggregate being skewed, and at least one 24-hour aggregate
skewed. The reason why I use the word skewed and not wrong is because the aggregates would
not be wrong, they would just be based on a limited amount of data. Furthermore, this also
impacts baseline computations since they are based on 1-hour aggregates. Also OOB alerting
will be impacted since the baselines will be calculated based on limited data. So false
alerts could be triggered or no alert at all be triggered due to baselines with small data
samples.
Now we considered running RHQ but with the aggregate service down. In this case RHQ could
continue to collect raw metrics. But that creates other problems. Since baselines are
based on 1h hour aggregates, baseline computations will have to turned of too. Since
baselines will not be computed until the migration is over then OOB alerts will not be
fired until all the data is migrated.
On the other hand, if the RHQ server is down then metrics will still be collected at agent
level and the server will get them after the upgrade. The advantage here is that migration
process will be faster since the relation database, Cassandra, and system resources (disk
and CPU) will be dedicated almost exclusively to the migration.
So, is it better to stop aggregation, baseline computations, and have no OOB alerts until
the migration is over knowing that the migration process might run longer because RHQ is
running? Or it is better to run a faster migration with RHQ server down?
On 1/14/2013 12:00 PM, Elias Ross wrote:
> Just my 2 cents:
>
> I would like to continue running RHQ with minimal downtime (less
> than
> 30 minutes if possible). The main issue with downtime is loss of
> systems monitoring capability. Historical data is definitely nice
> to
> have, but system monitoring trumps this. I would sooner drop the
> metrics data than have to face an unpredictable migration process.
>
> I would prefer if migration could happen concurrent to running of
> RHQ.
> _______________________________________________
> rhq-devel mailing list
> rhq-devel(a)lists.fedorahosted.org
>
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel