I've spent the day fixing various issues in qemu on Rawhide, but there's one remaining issue. The tests stubbornly fail on i686. It seems as if there is some issue with TLS (is gnutls borked on i686?)
https://koji.fedoraproject.org/koji/taskinfo?taskID=98075482 [Ignore the transient aarch64 failure ...]
Why are we building qemu on i686?
Although leaf packages can decide to drop i686 unilaterally, and I am sorely tempted, this is a critical package so I'd like to know if you have a valid use case for qemu, qemu-img etc on i686.
https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval
If we remove qemu it will likely have ripple effects through the rest of the virt stack, which we'll deal with (by excluding i686 on those too).
Rich.
On Mon, Feb 27, 2023 at 06:28:56PM +0000, Richard W.M. Jones wrote:
I've spent the day fixing various issues in qemu on Rawhide, but there's one remaining issue. The tests stubbornly fail on i686. It seems as if there is some issue with TLS (is gnutls borked on i686?)
https://koji.fedoraproject.org/koji/taskinfo?taskID=98075482 [Ignore the transient aarch64 failure ...]
Why are we building qemu on i686?
Although leaf packages can decide to drop i686 unilaterally, and I am sorely tempted, this is a critical package so I'd like to know if you have a valid use case for qemu, qemu-img etc on i686.
https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval
If we remove qemu it will likely have ripple effects through the rest of the virt stack, which we'll deal with (by excluding i686 on those too).
Furthermore if we drop i686 builds, then I'd suggest we also drop armv7 builds too. Neither arch provides installable media for Fedora anymore, and QEMU doesn't ship any libraries which would be multilib relevant. Even if there is a 32-bit library that is exec'ing qemu-img, that should be OK with a 64-bit qemu-img binary.
Upstream QEMU is actively discussing dropping 32-bit host support too.
With regards, Daniel
On Mon, Feb 27, 2023 at 7:36 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Mon, Feb 27, 2023 at 06:28:56PM +0000, Richard W.M. Jones wrote:
I've spent the day fixing various issues in qemu on Rawhide, but there's one remaining issue. The tests stubbornly fail on i686. It seems as if there is some issue with TLS (is gnutls borked on i686?)
https://koji.fedoraproject.org/koji/taskinfo?taskID=98075482 [Ignore the transient aarch64 failure ...]
Why are we building qemu on i686?
Although leaf packages can decide to drop i686 unilaterally, and I am sorely tempted, this is a critical package so I'd like to know if you have a valid use case for qemu, qemu-img etc on i686.
https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval
If we remove qemu it will likely have ripple effects through the rest of the virt stack, which we'll deal with (by excluding i686 on those too).
Furthermore if we drop i686 builds, then I'd suggest we also drop armv7 builds too. Neither arch provides installable media for Fedora anymore, and QEMU doesn't ship any libraries which would be multilib relevant. Even if there is a 32-bit library that is exec'ing qemu-img, that should be OK with a 64-bit qemu-img binary.
I'm not really sure which armv7hl builds you're talking about here? Support for armv7hl has been dropped entirely with Fedora 37: https://fedoraproject.org/wiki/Changes/RetireARMv7
Fabio
On Mon, Feb 27, 2023 at 07:43:58PM +0100, Fabio Valentini wrote:
On Mon, Feb 27, 2023 at 7:36 PM Daniel P. Berrangé berrange@redhat.com wrote:
On Mon, Feb 27, 2023 at 06:28:56PM +0000, Richard W.M. Jones wrote:
I've spent the day fixing various issues in qemu on Rawhide, but there's one remaining issue. The tests stubbornly fail on i686. It seems as if there is some issue with TLS (is gnutls borked on i686?)
https://koji.fedoraproject.org/koji/taskinfo?taskID=98075482 [Ignore the transient aarch64 failure ...]
Why are we building qemu on i686?
Although leaf packages can decide to drop i686 unilaterally, and I am sorely tempted, this is a critical package so I'd like to know if you have a valid use case for qemu, qemu-img etc on i686.
https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval
If we remove qemu it will likely have ripple effects through the rest of the virt stack, which we'll deal with (by excluding i686 on those too).
Furthermore if we drop i686 builds, then I'd suggest we also drop armv7 builds too. Neither arch provides installable media for Fedora anymore, and QEMU doesn't ship any libraries which would be multilib relevant. Even if there is a 32-bit library that is exec'ing qemu-img, that should be OK with a 64-bit qemu-img binary.
I'm not really sure which armv7hl builds you're talking about here? Support for armv7hl has been dropped entirely with Fedora 37: https://fedoraproject.org/wiki/Changes/RetireARMv7
Opps, yes, I'm getting my streams mixed up. Its only i686 that's special.
With regards, Daniel
[Only direct dependencies] Packages that BuildRequire qemu*:
[n] appliance-tools cloud-utils [n] cockpit-machines [n] containers-common copr-rpmbuild [n] diskimage-builder guestfs-tools [n] imagefactory-plugins kata-containers kiwi [n] kiwi-boxed-plugin [ns]koji libguestfs libvirt [n] libvirt-test-API [s] lorax [n] nodepool [n] openqa os-autoinst [n] osbuild [x] osbuild-composer [x] qemu-sanity-check qt-virt-manager [n] ravada [n] standard-test-roles [n] virt-backup virt-v2v virtiofsd xen
key: [n] = noarch [s] = dependency limited to a subpackage [x] = already excludearch i686
Packages that BuildRequire libvirt*:
[n] cockpit-machines [s] collectd condor [ns]fence-agents gnome-boxes guestfs-tools libguestfs libvirt-cim libvirt-dbus libvirt-designer libvirt-glib [n] libvirt-java libvirt-python libvirt-sandbox [n] libvirt-test-API [n] mom nbdkit ocaml-libvirt [n] oz [s] pcp perl-Sys-Virt perl-Sys-Virt-TCK php-libvirt qt-virt-manager [n] ravada rubygem-ruby-libvirt sblim-gather [s] systemtap [n] testcloud [n] tmt vhostmd virt-bootstrap [n] virt-lightning [n] virt-manager virt-top virt-v2v virt-viewer
It seems plausible that we would still ship libvirt.i686, but remove the qemu-related subpackages (leaving, essentially, only the libvirt client bindings). In which case the second list can be largely ignored.
Rich.
On Tue, Feb 28, 2023 at 10:52:06AM +0000, Richard W.M. Jones wrote:
[Only direct dependencies] Packages that BuildRequire qemu*:
[n] appliance-tools cloud-utils [n] cockpit-machines [n] containers-common copr-rpmbuild [n] diskimage-builder guestfs-tools [n] imagefactory-plugins kata-containers kiwi [n] kiwi-boxed-plugin [ns]koji libguestfs libvirt [n] libvirt-test-API [s] lorax [n] nodepool [n] openqa os-autoinst [n] osbuild [x] osbuild-composer [x] qemu-sanity-check qt-virt-manager [n] ravada [n] standard-test-roles [n] virt-backup virt-v2v virtiofsd xen
key: [n] = noarch [s] = dependency limited to a subpackage [x] = already excludearch i686
Packages that BuildRequire libvirt*:
[n] cockpit-machines [s] collectd condor [ns]fence-agents gnome-boxes guestfs-tools libguestfs libvirt-cim libvirt-dbus libvirt-designer libvirt-glib [n] libvirt-java libvirt-python libvirt-sandbox [n] libvirt-test-API [n] mom nbdkit ocaml-libvirt [n] oz [s] pcp perl-Sys-Virt perl-Sys-Virt-TCK php-libvirt qt-virt-manager [n] ravada rubygem-ruby-libvirt sblim-gather [s] systemtap [n] testcloud [n] tmt vhostmd virt-bootstrap [n] virt-lightning [n] virt-manager virt-top virt-v2v virt-viewer
It seems plausible that we would still ship libvirt.i686, but remove the qemu-related subpackages (leaving, essentially, only the libvirt client bindings). In which case the second list can be largely ignored.
Yes, my intent would be to disable build of the libvirt-daemon-driver-qemu, libvirt-daemon-qemu and libvirt-daemon-kvm sub-RPMS on i686. This would leave the libvirt clients and most expecially the -devel package. So I think we can ignore stuff with a BuildRequires on libvirt(-devel).
What would matter is anything with a Requires on the QEMU related libvirt sub-RPMs directly.
With regards, Daniel
On Mon, Feb 27, 2023 at 06:28:56PM +0000, Richard W.M. Jones wrote:
It seems as if there is some issue with TLS (is gnutls borked on i686?)
Yes as it happens, see the thread here:
https://sourceware.org/pipermail/libc-alpha/2023-March/145992.html
Rich.
On Thu, Mar 02, 2023 at 11:06:04AM +0000, Richard W.M. Jones wrote:
On Mon, Feb 27, 2023 at 06:28:56PM +0000, Richard W.M. Jones wrote:
It seems as if there is some issue with TLS (is gnutls borked on i686?)
Yes as it happens, see the thread here:
https://sourceware.org/pipermail/libc-alpha/2023-March/145992.html
I've filed an upstream bug against gnutls too
https://gitlab.com/gnutls/gnutls/-/issues/1466
it isn't really GNUTLS fault - it has had the time_t ABI breakage imposed on it by an well meaning change in GNULIB that has had undesirable consequences.
My suggested workaround in Fedora is --disable-year2038
https://src.fedoraproject.org/rpms/gnutls/pull-request/78
which is what the Fedora wget maintainer has also done for thue same time_t ABI comapt problem.
If any other Fedora maintainer looks after packages that use GNULIB, then it is quite likely that they also need to add --disable-year2038 to their specfiles to avoid i686 brokenness when talking to any non-glibc libraries that use time_t, or exposing a library using time_t
With regards, Daniel
On Thu, Mar 02, 2023 at 11:57:42AM +0000, Daniel P. Berrangé wrote:
If any other Fedora maintainer looks after packages that use GNULIB, then it is quite likely that they also need to add --disable-year2038 to their specfiles to avoid i686 brokenness when talking to any non-glibc libraries that use time_t, or exposing a library using time_t
There are about 150 packages that provide "bundled(gnulib)". About 50 have "lib" in their name. Just wondering, wouldn't it be less work to enable 64-bit time_t in the rpm CFLAGS, rebuild everything, and deal with a potentially smaller number of broken packages with the benefit of fixing Y2038?
On Thu, Mar 02, 2023 at 02:35:11PM +0100, Miroslav Lichvar wrote:
On Thu, Mar 02, 2023 at 11:57:42AM +0000, Daniel P. Berrangé wrote:
If any other Fedora maintainer looks after packages that use GNULIB, then it is quite likely that they also need to add --disable-year2038 to their specfiles to avoid i686 brokenness when talking to any non-glibc libraries that use time_t, or exposing a library using time_t
There are about 150 packages that provide "bundled(gnulib)". About 50 have "lib" in their name. Just wondering, wouldn't it be less work to enable 64-bit time_t in the rpm CFLAGS, rebuild everything, and deal with a potentially smaller number of broken packages with the benefit of fixing Y2038?
The main reason for i686 continuing to exist at all in Fedora is to facilitate running of externally distributed 32-bit apps (for example wine/steam related). Those apps are all going to be built with 32-bit time_t.
IOW if Fedora unconditionally rebuilt its world with 64-bit time_t, it would be killing ABI compatibility with the very things that its i686 RPMs aim to support.
AFAICT, Fedora either keeps 32-bit time_t, or kills i686 for good.
With regards, Daniel
[1] https://fedoraproject.org/wiki/Changes/EncourageI686LeafRemoval
"The remaining use cases of i686 packages (i.e. "multilib") for popular 32-bit applications should continue to work. For example, installing the Steam client (steam.i686), Wine, or other applications that require 32-bit compatibility libraries should still be possible, and not fail due to broken dependencies. "
Daniel P. Berrangé wrote:
The main reason for i686 continuing to exist at all in Fedora is to facilitate running of externally distributed 32-bit apps (for example wine/steam related). Those apps are all going to be built with 32-bit time_t.
WINE is packaged in Fedora. If Fedora rebuilds everything with 64-bit time_t, the packaged WINE will also be rebuilt. And any Windows applications run in WINE should not be seeing the underlying time_t at all, only the Windows or crtdll/msvcrt/ucrt time APIs.
Kevin Kofler
* Daniel P. Berrangé:
AFAICT, Fedora either keeps 32-bit time_t, or kills i686 for good.
I agree with this position.
For additional details, here is what I wrote in the other thread (regarding defaulting time64):
| That still needs some per-package work (mainly for scripting languages | using FFI) because dlsym for gettimeofday etc. still find the 32-bit | variant. There are various ways we can hack around that, I guess. | | Anyway, this dual ABI break (for off_t and time_t) needs to be proposed | as a Fedora change, and we can discuss mechanics if Fedora wants to move | in that direction. I think this is far from a given because a | still-unknown amount of third-party software will break. GNUTLS, for | example, used to have a fairly stable ABI: libgnutls.so.30 goes back a | couple of years; I think it was part of CentOS 7 already. | | I think the first step is to decide if we want to do this. After that, | we can discuss mechanics. For example, traditionally, ABI changes like | this have not been implemented through build flags injection in Fedora, | rather we updated the toolchain defaults. | | Needless to say, I have very little interest to work on this (I consider | all this a pointless distraction, to be blunt), but I guess I can help | with toolchain enablement.
Thanks, Florian
On Fri, Mar 03, 2023 at 12:56:27PM +0100, Florian Weimer wrote:
- Daniel P. Berrangé:
AFAICT, Fedora either keeps 32-bit time_t, or kills i686 for good.
I agree with this position.
For additional details, here is what I wrote in the other thread (regarding defaulting time64):
| That still needs some per-package work (mainly for scripting languages | using FFI) because dlsym for gettimeofday etc. still find the 32-bit | variant. There are various ways we can hack around that, I guess. | | Anyway, this dual ABI break (for off_t and time_t) needs to be proposed | as a Fedora change, and we can discuss mechanics if Fedora wants to move | in that direction. I think this is far from a given because a | still-unknown amount of third-party software will break. GNUTLS, for | example, used to have a fairly stable ABI: libgnutls.so.30 goes back a | couple of years; I think it was part of CentOS 7 already. | | I think the first step is to decide if we want to do this. After that, | we can discuss mechanics. For example, traditionally, ABI changes like | this have not been implemented through build flags injection in Fedora, | rather we updated the toolchain defaults. | | Needless to say, I have very little interest to work on this (I consider | all this a pointless distraction, to be blunt), but I guess I can help | with toolchain enablement.
Yes, this whole episode makes me want to kill off i686 builds of everything I maintain, even when its not a leaf package. I only spent time investigating the root cause in gnutls because I saw valgrind warnings and wanted to be sure there wasn't a security flaw lurking that would impact x86_64 too. I have absolutely no interest in spending resources to retrofit 64-bit time_t to a build target that ceased to be relevant to the overwhealming majority of Fedora users many years ago.
With regards, Daniel
On 03/03/2023 13:03, Daniel P. Berrangé wrote:
Yes, this whole episode makes me want to kill off i686 builds of everything I maintain, even when its not a leaf package.
Me too. It worth the F39 System-Wide proposal: drop all i686 builds that are not needed by Wine/Steam.
On Fri, Mar 3 2023 at 12:03:12 PM +0000, Daniel P. Berrangé berrange@redhat.com wrote:
Yes, this whole episode makes me want to kill off i686 builds of everything I maintain, even when its not a leaf package. I only spent time investigating the root cause in gnutls because I saw valgrind warnings and wanted to be sure there wasn't a security flaw lurking that would impact x86_64 too. I have absolutely no interest in spending resources to retrofit 64-bit time_t to a build target that ceased to be relevant to the overwhealming majority of Fedora users many years ago.
If the i686 builds only exist for Steam and Wine, can we say users should switch to containerized distributions instead (e.g. flatpak, snap, or whatever)? Does it really makes sense to build all the various dependencies of these applications in an age where containers exist?
On Fri, Mar 3, 2023 at 2:34 PM Michael Catanzaro mcatanzaro@redhat.com wrote:
On Fri, Mar 3 2023 at 12:03:12 PM +0000, Daniel P. Berrangé berrange@redhat.com wrote:
Yes, this whole episode makes me want to kill off i686 builds of everything I maintain, even when its not a leaf package. I only spent time investigating the root cause in gnutls because I saw valgrind warnings and wanted to be sure there wasn't a security flaw lurking that would impact x86_64 too. I have absolutely no interest in spending resources to retrofit 64-bit time_t to a build target that ceased to be relevant to the overwhealming majority of Fedora users many years ago.
If the i686 builds only exist for Steam and Wine, can we say users should switch to containerized distributions instead (e.g. flatpak, snap, or whatever)? Does it really makes sense to build all the various dependencies of these applications in an age where containers exist?
While I agree to some extent (the "unofficial" Steam flatpak from flathub works quite well), I don't think dropping wine is a good idea. As far as I know, it's used to run tests in some MinGW packages.
Fabio
* Michael Catanzaro:
If the i686 builds only exist for Steam and Wine, can we say users should switch to containerized distributions instead (e.g. flatpak, snap, or whatever)?
Is Valve already maintaining a 32-bit distribution? If not, where would their Flatpak get the builds from?
And I thought Wine can run 32-bit Windows applications against 64-bit system libraries these days? Or at least it's getting there. Then Wine wouldn't be a reason to keep 32-bit builds around.
| *** WoW64 | | - The 64-bit Windows-on-Windows (WoW64) architecture is implemented, and | supports running a 32-bit Windows application inside a 64-bit Unix host | process, using thunks to map 32-bit NT system calls to the 64-bit NTDLL. | | - WoW64 thunks are implemented for most Unix libraries, enabling a 32-bit PE | module to call a 64-bit Unix library. Once the remaining modules are | converted to PE, this will make it possible to run 32-bit applications | without installing 32-bit Unix libraries.
https://www.winehq.org/announce/7.0
Thanks, Florian
On Fri, Mar 3 2023 at 03:21:33 PM +0100, Florian Weimer fweimer@redhat.com wrote:
Is Valve already maintaining a 32-bit distribution? If not, where would their Flatpak get the builds from?
There is already a Steam flatpak based on freedesktop-sdk, which builds just enough i686 stuff for Steam to work.
And I thought Wine can run 32-bit Windows applications against 64-bit system libraries these days? Or at least it's getting there. Then Wine wouldn't be a reason to keep 32-bit builds around.
| *** WoW64 | | - The 64-bit Windows-on-Windows (WoW64) architecture is implemented, and | supports running a 32-bit Windows application inside a 64-bit Unix host | process, using thunks to map 32-bit NT system calls to the 64-bit NTDLL. | | - WoW64 thunks are implemented for most Unix libraries, enabling a 32-bit PE | module to call a 64-bit Unix library. Once the remaining modules are | converted to PE, this will make it possible to run 32-bit applications | without installing 32-bit Unix libraries.
Very interesting.
Florian Weimer wrote:
And I thought Wine can run 32-bit Windows applications against 64-bit system libraries these days? Or at least it's getting there. Then Wine wouldn't be a reason to keep 32-bit builds around.
| *** WoW64 | | - The 64-bit Windows-on-Windows (WoW64) architecture is implemented, and | supports running a 32-bit Windows application inside a 64-bit Unix | host process, using thunks to map 32-bit NT system calls to the 64-bit | NTDLL. | | - WoW64 thunks are implemented for most Unix libraries, enabling a | 32-bit PE module to call a 64-bit Unix library. Once the remaining | modules are converted to PE, this will make it possible to run 32-bit | applications without installing 32-bit Unix libraries.
"Once the remaining modules are converted to PE" should be now:
https://www.winehq.org/news/2023012401 | Wine 8.0 Released | January 24, 2023 [snip] | The main achievement is the completion of the conversion to PE format.
Kevin Kofler
On Thu, Mar 02, 2023 at 02:35:11PM +0100, Miroslav Lichvar wrote:
On Thu, Mar 02, 2023 at 11:57:42AM +0000, Daniel P. Berrangé wrote:
If any other Fedora maintainer looks after packages that use GNULIB, then it is quite likely that they also need to add --disable-year2038 to their specfiles to avoid i686 brokenness when talking to any non-glibc libraries that use time_t, or exposing a library using time_t
There are about 150 packages that provide "bundled(gnulib)". About 50 have "lib" in their name. Just wondering, wouldn't it be less work to enable 64-bit time_t in the rpm CFLAGS, rebuild everything, and deal with a potentially smaller number of broken packages with the benefit of fixing Y2038?
So a few facts first:
(1) Gnulib, in some common circumstances, sets -D_TIME_BITS=64
(2) Programs might also choose to set -D_TIME_BITS=64, even if they don't use Gnulib.
(3) This flag only affects 32 bit platforms like i686 and armv7, changing time_t from 32 to 64 bit.
(4) Also this only matters if time_t leaks into header files, because it could cause the library's ABI to change. If time_t is used entirely internal to the program/library, it shouldn't be a problem.
(5) A problem arises when the caller and callee disagree on the size of time_t.
Now, as Dan points out in the linked thread on libc-alpha, the reason we have i686 in Fedora at all is to support third party programs like Wine and Steam. Those may still be using 32-bit time_t for a while. That's an argument for sticking with the current situation.
This might change in future -- eg. if these external programs are being compiled on, say, Debian, and Debian were to switch to using -D_TIME_BITS=64 everywhere, and we found more programs expecting it, then we'd have to switch. You can see why this is going to be problematic for the period where there are mixed old & new binaries.
Also libraries that do expose raw time_t in header files are actually broken. Really they should expose an explicit 64 bit type (eg. int64_t) and converting to time_t internally. But probably that ship has sailed.
Rich.