Hello Everybody,
After a few weeks of slowly collecting data and then refining and optimizing the import
code I have some good numbers to share.
For this exercise I created a tool [1] that generates random data based on the estimates
I've previously done [2]. I used two models 1)everything local (Cassandra, migration
tool, postgres database) and 2)remote postgres database but local Cassandra and migration
tool. With the remote postgres, I had a somewhat slow network connection to the server but
the hardware was much better than I had locally.
The migration tools works in two modes:
1) migrated everything, then delete everything at once
2) migrated a little bit of data, then delete what got migrated and repeat
Here is what I can report so far:
1) Test data creation is resource intensive but the real bottleneck is the SQL insert
process. The time required to insert test data for 1 agent slowly increases over time.
When I used the remote postgres server, the standard deviation to insert data for 1 agent
was low. So the insert process slows down less with better hardware (expected). Some
telemetry for this can be found at [3].
2) The observations from test data generation are very useful in case we decide to make
the gradual data migration default (the second migration process). Deleting migrated data
can consume a lot of time for the initial batches but decrease over time as there will
less data left in the database.
3) Average time to migrate data for 10 agents is between 20 to 27 minutes with everything
local and data deletion at the end. The CPU utilization is high (90% or above) because
Cassandra, the migration tool and postgres are all crunching numbers.
4) Average time to migrate 10 agents is between 35-45 minutes with a remote SQL and data
deletion at the end. I attribute this difference on the slow connection between my box and
the remote postgres server. The CPU utilization was lower (50-60%) which means the
bottleneck was the network.
5) The time required for migration does not scale linearly. Migrating 20 agents will take
slightly longer than two independent 10 agent migrations. Because of the indexing on
Cassandra side, more data added means more indexing workload.
6) The average time required to migrate 40 agents is between 1.5hours and 2 hours with
everything local. Still running tests with the remote postgres server for 40 agents.
7) I am still gathering telemetry for migrating 125 agents but some of the early runs were
north of 7.5 hours. The code has received many optimizations so that time is be lower now
but not half.
8) With everything local, I could see about 1 to 1.2 million raw metrics records per
minute migrated to Cassandra.
9) The amount of memory required by the migration tool is around 700 MB. That is mainly
because the migration tool processes a batch of 30K records from SQL at once.
10) If I continue to optimize the import tool I do not expect to reduce the total amount
of time by more than 25% because of the large of amount data/processing required. Data
flows from postgres, to the migration tool, and into Cassandra. At each step there is
substantial processing; for example, the migration tool needs to convert the data from the
old model into the new model.
So based on these numbers, the migration would take a few hours for large deployments. If
we choose to do the migration with the RHQ server down then there will be no RHQ UI
capability for a few hours; metrics will still be collected by the agents but not reported
to the server for a few hours. Is that acceptable?
On the other hand (as discussed in previous emails), if migration is happening with the
server up then aggregates will be skewed until the full data is migrated. Also, if the RHQ
server is running, the migration process will need a little slowdown so it does not gobble
all the server CPU. Is this better?
Any thoughts?
Thank you,
Stefan Negrea
Software Engineer
[1]
http://git.fedorahosted.org/cgit/rhq/rhq.git/tree/modules/enterprise/serv...
[2]
https://docs.jboss.org/author/display/RHQ/Metrics+Data+Migration+-+Design
[3]
https://docs.jboss.org/author/download/attachments/56885438/GenerationTel...