Dne 29. 05. 19 v 23:52 Josh Boyer napsal(a):
On Wed, May 29, 2019 at 4:20 PM Ben Cotton <bcotton(a)redhat.com>
wrote:
>
>
https://fedoraproject.org/wiki/Changes/Switch_RPMs_to_zstd_compression
>
> = Switch RPMs to zstd compression =
>
> == Summary ==
> Binary RPMs are currently compressed with xz level 2.
> Switching to zstd would increase decompression speed significantly.
>
> == Owner ==
> * Name: [[User:dmach| Daniel Mach]]
> * Email: dmach(a)redhat.com
>
> == Detailed Description ==
> * The change requires setting a new compression algorithm in rpm
> macros. Then a mass rebuild of all packages is required.
The gcc team often does mass rebuilds on the side prior to updating
gcc in Fedora. Would it be possible to do the same or leverage their
rebuild work with the default changed in RPM to see what the true
overall savings is? That would get us a lot more data to see if it's
truly going to benefit the distro in terms of size and installation
speed.
I rebuilt the packages that are available in fedora:30 docker image:
https://copr.fedorainfracloud.org/coprs/dmach/fedora-zstd/
The overall size is roughly equal to xz compressed RPMs.
It's not comparable with the whole Fedora repo, but it's a good start.
I can build more packages if a larger sample is needed.
> * The macro for setting the compression is: %define _binary_payload w19.zstdio
> * The recommended compression level is 19. The builds will take
> longer, but the additional compression time is negligible in the total
> build time and it pays off in better compression ratio than xz lvl2
> has.
> * SRPM payload compression should stay at gzip (there's almost no
> benefit in changing the compression, because SRPM's contents is
> compressed already)
>
> === Use case: Firefox installation ===
> I rebuilt firefox-66.0.5-1.fc30 with zstd level19.
> Then I compared installation times with the original (xz compressed) package:
>
> {| class="wikitable"
> |-
> ! Compression !! Target File System !! Time
> |-
> | xz level 2 || tmpfs || 8s
> |-
> | xz level 2 || ext4 on nvme || 11s
> |-
> | zstd level 19 || tmpfs || 2s
> |-
> | zstd level 19 || ext4 on nvme || 4s
> |-
> |}
>
>
> === Comparison of compression algorithms and levels ===
> Following table shows '''cpio''' and
'''compressed cpio''' extraction
> times into a tmpfs. Actual times in decompressing RPMs will differ due
> to extracting on an actual disk and also some overhead in the RPM tool
> (checks, scriptlets).
>
> {| class="wikitable"
> |-
> ! Compression !! Level !! Size B !! Size GiB
> !! Compression time !! Compression time, 4 threads !!
> Decompression time !! Comment
> |-
> | CPIO || - || 5016785692 || 4,7
> || - || - || -
> ||
> |-
> | xz || 2 || 1615017616 || 1,6
> || 9m55s || - || 1m36s
> || slow decompression
> |-
> | pxz || 2 || 1631869880 || 1,6
> || - || 6m11s || 1m38s
> || slow decompression
> |-
> | gzip || 9 || 2086354992 || 2,0
> || 10m23s || - || 31s
> || insufficient compression ratio
> |-
> | bzip2 || 9 || 1889161565 || 1,8
> || 8m || - || 2m50s
> || very slow decompression; compression ratio could be
> better
> |-
> | zstd || 3 || 1913536587 || 1,8
> || 31s || 29s || 6,5s
> ||
> |-
> | zstd || 10 || 1737928978 || 1,7
> || 3m27s || 2m34s || 6,3s
> ||
> |-
> | zstd || 15 || 1717303256 || 1,7
> || 9m37s || 6m34s || 6,3s
> || identical compression speed to xz; fast decompression;
> slightly worse compression ratio than xz
> |-
> | zstd || 17 || 1635525492 || 1,6
> || 16m16s || 11m20s || 6,7s
> ||
> |-
> | zstd || 19 || 1575843696 || 1,5
> || 24m2s || 18m55s || 7,7s
> ||
> |-
> |}
>
> == Benefit to Fedora ==
> * Faster installations/upgrades of user systems
> * Faster koji builds (installations in build roots)
> * Faster container builds
> * Lower bandwidth on mirrors if we choose the highest compression level
>
> == Scope ==
> * Proposal owners: submit a patch to redhat-rpm-config
> * Other developers: redhat-rpm-config maintainer: include the patch
> and make a new build
> * Release engineering: [
https://pagure.io/releng/issue/8345 #8345]
> mass rebuild is needed
>
> == Upgrade/compatibility impact ==
> * RPM in Fedora supports zstd compression already (from Fedora 28,
> rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected.
> * Fedora <= 27 and some other distros will not be able to decompress
> zstd-compressed RPMs.
If we did this, wouldn't it make it very difficult to use tools like
mock on RHEL / CentOS 7 to build for Fedora 3x? Or does RHEL 7 RPM
support zstd?
This is a good point.
I hope that mock's bootstrap feature could cover that, but I need to
verify that.
Does MBS's concept of platform modules help us build a module
across
the RPM zstd-support boundary? I think it does, but I honestly can't
remember for sure and I'm not aware of the details that go into MBS
performing the build.
I'm afraid I can't answer it because of lack of
knowledge how MBS works.
> == How To Test ==
> * dnf install <package>
> * rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
> * expected output: zstd 19
>
> Also the overall system installation time should decrease significantly.
>
> == User Experience ==
> See '''Benefit to Fedora'''
>
> == Dependencies ==
> N/A
>
> == Contingency Plan ==
> * Contingency mechanism: Not needed, Fedora will stay at current compression.
> * Contingency deadline: N/A
This seems wrong. If we get through a mass rebuild (or partial mass
rebuild) and find some ugly unknown issue with zstd compression, we're
going to have to do another mass rebuild to revert everything back,
correct? That should be listed as the Contingency, even if it's
unlikely.
ACK. Will update the Change page accordingly.
> * Blocks release? No
Are we not advocating for a fully successful mass rebuild? Would we
ship the distribution with only a portion (significant or otherwise)
switched to zstd?
I did not consider the mass rebuild a blocker (I was mainly
thinking
about the feature itself becoming a blocker). I'll change the Change
page accordingly.
>
>> * Blocks product? N/A
>>
>> == Documentation ==
>> N/A
>>
>> == Release Notes ==
>> RPMs have switched to zstd compression level 19.
>> Users will benefit from faster package decompression.
>> Users that build their packages will experience slightly longer build times.
>>
>> --
>> Ben Cotton
>> He / Him / His
>> Fedora Program Manager
>> Red Hat
>> TZ=America/Indiana/Indianapolis
>> _______________________________________________
>> devel mailing list -- devel(a)lists.fedoraproject.org
>> To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org
>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
>> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org