Hello all,
TL;DR: casync is an interesting piece of technology, but doesn't work for our
current test images.
As I just merged another fedora-26 test image refresh, I took some time to
evaluate casync [1]. On my laptop I still have the old image (c75882fd), and on
my colo server (which has a really fast internet connection) I downloaded the
new image (554546f5).
The main tradeoff is the chunk size: smaller sizes increase the likelihood of a
particular chunk already being present locally, but increase the number chunks
that need to be downloaded. As this usually happens through HTTP, lots of small
files kill performance completely (mostly due to the usual TCP slow start
behaviour, but also due to the extra effort of making the connection itself).
The reference time on my VDSL-25:
$ time bots/image-download fedora-26
real 9m52.313s
This is still bearable, but it would be even more interesting to improve image
uploads (they take about 45 mins here).
First I created a casync index from the fedora 26 qcow image on my server, with
the the default option (64 kB chunk size):
server$ cd /path/to/testdir
server$ casync make fedora-26.caibx images/fedora-26-*.qcow2
Now on my laptop I use casync to get the new image, using as many chunks from
the previous one as a basis, plus the new chunks from my server:
$ casync extract --seed images/fedora-26-c75882*.qcow2
http://piware.de/tmp/cockpit-images-casync/fedora-26.caibx
images/fedora-26-554546f5c12141c3bc2e92ba53564ad490f6974aea5f7df831ebcab563bf1b62.qcow2
This has to download thousands of chunks, each using one HTTP request: It took
slightly over 22 minutes. Oddly enough this didn't save the downloaded chunks
anywhere, so I'm unable to tell whether this actually saved any bandwidth.
Next I changed the average chunk size to 8 MB, and produced a new chunk store:
server$ casync --store 8m.castr --chunk-size=8M make fedora-26-8mb.caibx
images/fedora-26-*.qcow2
Trying to download it fails:
$ casync -vv --store
http://piware.de/tmp/cockpit-images-casync/8m.castr --seed
images/fedora-26-c75882fdee8847229eb89707d41b4488286605fc9f150811a5d950b8a1399825.qcow2
extract
http://piware.de/tmp/cockpit-images-casync/fedora-26-8mb.caibx
images/fedora-26-554546f5c12141c3bc2e92ba53564ad490f6974aea5f7df831ebcab563bf1b62.qcow2
[...]
Chunk too large
Failed to acquire
http://piware.de/tmp/cockpit-images-casync/8m.castr/6265/626515b391d445ef...
Failed to run synchronizer: Broken pipe
OK, fair enough. Next attempt with 1MB worked, but took pretty much exactly the
same time as bots/image-download. Apparently there wasn't any actual saving,
the qcow images seem to differ too much in between image rebuilds.
Cross-checking with rsync over ssh (against the old image):
| 1,515,291,648 100% 2.68MB/s 0:09:00 (xfr#1, to-chk=0/1)
|
| sent 282,439 bytes received 1,479,233,072 bytes 2,712,219.09 bytes/sec
| total size is 1,515,291,648 speedup is 1.02
Maybe there is some trick ("sort" the blocks on the file system or so) to make
qcow files rsyncable better. But in conclusion, right now this delta approach
doesn't buy us much.
Martin
[1]
http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-image...