On Thu, May 30, 2019 at 11:44:23AM -0700, Samuel Sieb wrote:
On 5/30/19 7:54 AM, Daniel Mach wrote:
>>I think before approving such changes, owners need to do mass
>>rebuilds on their own and provide a graph of changes in size
>>between original compression format and new one(s).
>Doing that equals to the mass rebuild.
>I'd rather do an analysis and if the numbers look sane, I'd prefer
>doing a mass rebuild in a side tag so we can use the builds if the
>results are sane. Hammering koji with so many scratch builds
>doesn't sound right to me.
Is it possible to just recompress the rpms instead of doing a full build?
+1, we
can certainly get approximate statistics without rebuilding packages.
I did a somewhat unscientific test with 1GB of packages from a mock cache:
$ rm -rf zstd; mkdir zstd; for i in
/var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/*rpm; do
rpm2cpio $i | zstd >zstd/$(basename $i .rpm).cpio.zstd -19 -;done
$ rm -rf xz; mkdir xz; for i in
/var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/*rpm; do
rpm2cpio $i | xz >xz/$(basename $i .rpm).cpio.xz -2 -;done
$ rm -rf zstd20; mkdir zstd20; time for i in zstd/*; do zstdcat $i | zstd
>zstd20/$(basename $i) --ultra -20 -;done
$ du -sh /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/
/tmp/{xz,zstd,zstd20}
1019M /var/cache/mock/fedora-rawhide-x86_64/dnf_cache/fedora-2d95c80a1fa0a67d/packages/
985M /tmp/xz
946M /tmp/zstd
930M /tmp/zstd20
$ time xzcat /tmp/xz/* >/dev/null
xzcat /tmp/xz/* > /dev/null 78.45s user 0.78s system 99% cpu 1:19.72 total
$ time zstdcat /tmp/zstd/* >/dev/null
zstdcat /tmp/zstd/* > /dev/null 9.19s user 0.44s system 98% cpu 9.751 total
$ time zstdcat /tmp/zstd20/* >/dev/null
zstdcat /tmp/zstd20/* > /dev/null 8.82s user 0.50s system 99% cpu 9.394 total
Notes:
- this is all single-threaded, since rpm doesn't do multithreading.
I saw some discussion that multithreading is easier with zstd because it's
reproducible with zstd, but not entirely with xz. Enabling multithreading
would be very beneficial for compression.
- $ rpm -q zstd xz
zstd-1.4.0-1.fc29.x86_64
xz-5.2.4-3.fc29.x86_64
- machine:
$ lscpu
Architecture: x86_64
CPU(s): 12
...
NUMA node(s): 2
Model name: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
/var/ is on spinning rust, /tmp is RAM.
- I'm not providing timings of compression, because I only did one run and
this might not be reproducible. Unscientific impression was that zstd
was quite a bit slower when compressing.
Zbyszek