Cassandra - Data Migration

Heiko W.Rupp hrupp at redhat.com
Thu Mar 14 11:42:52 UTC 2013


Stefan,

thanks for sharing that numbers and experience.

Postgres allows for some speedup in inserting data, but obviously that is not the part we
need to optimize.

- would it be feasible to e.g. dump the content of postgres into a "text file" and feed this
into cassandra for import? That would eliminate the jdbc-roundtripping and could even
allow to fill cassandra offline (while having the server work against postgres). If that
offline dump is in cassandra, the tool would only read the data from the time the above
snapshot was taken and transfer it over (not needed to total offline migration).

In postgres one of the tricks to speed up imports is to use the "copy" command

postgres=# \h copy
Command:     COPY
Description: copy data between a file and a table

for this.

Other tricks are to either use no transactions at all or to use very large
transactions for the inserts and not per row.
Also for mass-data inserts it helps to have no indexes on the table and only 
add them after all the data is inserted.

I don't know if there is something similar in Cassandra though

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 203 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.fedorahosted.org/pipermail/rhq-devel/attachments/20130314/201a002e/attachment.sig>


More information about the rhq-devel mailing list