One practical example - the process of creating an OCI container image of the Fedora Flatpak runtime does a *lot* of zlib compression and decompression. (https://pagure.io/flatpak-module-tools/issue/22). A lot of that is temporary to save disk space and IO bandwidth and could be switched to zstd or a lower zlib compression level, but without any changes, dropping in zlib-ng for the system zlib on my system reduced the wall time for building the runtime from 9m13s to 6m49s. I suspect a lot of developer tasks would get noticeably faster with a faster zlib, if not by that margin.

Regards,
Owen


On Sat, Aug 5, 2023 at 11:12 AM Richard W.M. Jones <rjones@redhat.com> wrote:
The background to this is I've spent far too long trying to optimize
the conversion of qcow2 files to raw files.  Most existing qcow2 files
that you can find online are zlib compressed, including the qcow2
images provided by Fedora.  Each cluster in the file is separately
compressed as a zlib stream, and qemu uses zlib library functions to
decompress them.  When downloading and decompressing these files, I
measured 40%+ of the total CPU time is doing zlib decompression.

[You don't need to tell me how great Zstd is, qcow2 supports this for
compression also, but it is not widely used by existing content.]

I was looking around for alternative zlib implementations and there
are several.  This AWS blog is a decent summary:

  https://aws.amazon.com/blogs/opensource/improving-zlib-cloudflare-and-comparing-performance-with-other-zlib-forks/

We already package zlib-ng in Fedora:

  https://src.fedoraproject.org/rpms/zlib-ng/blob/rawhide/f/zlib-ng.spec
  https://github.com/zlib-ng/zlib-ng

In my test zlib-ng is about 40% faster.

Sadly zlib-ng is not compiled with the ZLIB_COMPAT option.  What this
means in practice is that the zlib functions have different names
(eg. zng_inflateInit instead of inflateInit).  It is not a drop-in
replacement for zlib and software would need to be adjusted to use it.

However there is this bug / RFE to package the compat library.

  https://bugzilla.redhat.com/show_bug.cgi?id=2145239

It literally replaces /usr/lib64/libz.so.1, so pretty high stakes.  It
has the appropriate Provides/Conflicts.

Anyway, 40% (or whatever, but significant) performance improvement
sounds good for a very widely used operation.

So I'd like to ask Fedora ...

What do we think about the opt in approach of adding a patch similar
to the one proposed in bug 2145239, where I think you could "simply"
install zlib-ng to get better performance with existing software?
("simply" because it seems high risk of going wrong)

What about replacing zlib with zlib-ng?

Other ideas ...?  Does anyone know what other distros are doing?

Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue