Metrics Migration Tool - Cassandra

Stefan Negrea snegrea at redhat.com
Mon Jan 14 14:10:01 UTC 2013



----- Original Message -----
> From: "Michael Foley" <mfoley at redhat.com>
> To: rhq-users at lists.fedorahosted.org
> Cc: rhq-devel at lists.fedorahosted.org
> Sent: Thursday, January 10, 2013 11:13:59 AM
> Subject: Re: Metrics Migration Tool - Cassandra
> 
> 
> 
> Just the voice of QE on this .... I am hearing lots of discussion on
> design decisions on this data migration tool ... which is good.
> 
> 
> One word of caution (and I hear this in all your emails ... so I know
> you are thinking about this...this is just a reminder): Avoid
> premature optimization.
> 

Totally agree with you Mike. Premature optimization can be the root of a lot of evil. However, in this case I think the expectation was/is that the amount of data to be migrated is large. The only questions not addressed were "How large? How much data?" Hopefully I shed some light into this with with the updates to the design wiki:
https://docs.jboss.org/author/display/RHQ/Metrics+Data+Migration+-+Design

> 
> The course of action that makes sense to me:
> 
> 
> #1. Baseline the current implementation. Use small and large datasets
> typical of what a customer would use. Document the baseline. This
> will be incredibly useful moving forward to compare alternative
> solutions. You need the baseline.
> #2. Determine if any design optimizations need to be made.
> #3. Compare alternative solutions against the baseline.
> 
> 
> 
> 
> I guess what I am saying ... I think we need a baseline before this
> conversation can continue in a meaningful way.


I am not sure we can ever have exact figures for the amount of data to be migrated. Metrics collection is highly configurable in RHQ. Users can increase/decrease collection frequency of any metric. Another example is exact configuration of the resource monitored; a deployment with a single JBoss AS server but with 1000 applications inside will have a lot more metrics than the average JBoss AS install. The number of installed plugins and discovered resources is yet another example.


I hope that the estimates [1] (please see the estimation process) gives as stable starting point for creating test scenarios and some sort of timing baseline. However, I caution against treating these estimates as 99.9999% accurate because of highly configurable nature of metrics collection.


[1] https://docs.jboss.org/author/display/RHQ/Metrics+Data+Migration+-+Design


Thank you,
Stefan Negrea

> 
> 
> How can QE help establish a meaningful baseline? What do we need?
> Resources in the bladecenter? Determining an SLA or acceptance
> criteria for performance? Datasets?
> 
> 
> 
> 
> 
> 
> 
> From: "Thomas Segismont" <tsegismo at redhat.com>
> To: rhq-devel at lists.fedorahosted.org,
> rhq-users at lists.fedorahosted.org
> Sent: Thursday, January 10, 2013 12:02:07 PM
> Subject: Re: Metrics Migration Tool - Cassandra
> 
> Le 09/01/2013 20:32, John Sanda a écrit :
> > At this point, all we can do is speculate about how long the
> > migration will actually take until we do some load testing. If we
> > find that the migration is taking longer than we would like,
> > another option could be to explore using the bulk import/export
> > utilities provided by each of the databases.
> 
> I think working on bulk export files would be far more efficient. And
> it
> shouldn't be too difficult given the measurement tables have very
> simple
> schema (migrating to Cassandra may not be as simple as migrating
> these
> tables data though).
> 
> So why not having the two mechanisms:
> 1. batching with Hibernate which would support a larger number of
> deployments (Postgres, Oracle, SQLServer)
> 2. batching with bulk export files for the supported databases
> (Postgres, Oracle)
> 
> I know it's double code, test and support but I really doubt #1 can
> handle large amounts of data in less than a few hours.
> 
> And you're right, we cannot speculate on this and I don't believe we
> could make a release without actually trying the tool on different
> workloads.
> 
> Thomas
> _______________________________________________
> rhq-users mailing list
> rhq-users at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-users
> 
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list