On Sat, Aug 01, 2020 at 02:03:40PM -0600, Jeff Law wrote:
On Sat, 2020-08-01 at 12:12 +0200, Kevin Kofler wrote:
> seeing the amount of fallout from LTO, I really think that this feature
> ought to be dropped from F33, and evaluated carefully for F34 (i.e., can it
> be done without breaking the build of or miscompiling a large part of the
> distribution, once the bugs such as the ld bug discussed in this thread are
> fixed, or is it just unsafe to enable by default to begin with?). I.e.,
> revert it for F33 for sure, then decide whether to retry it for F34 or can
> it permanently.
Most of the fallout has been Nick pushing through binutils builds that are
broken. Seriously, there's been at least 4 builds pushed through that kept
bringing back the *same* problem.
And just to be clear, this has been 6+ months of behind the scenes work to find
and identify issues, fix broken packages, put global mitigations of broken crap
in place in place, opt-out packages that do things that are fundamentally
incompatible with LTO, etc. In fact it was that behind-the-scenes work that
pushed this feature from F32 to F33 as it just wasn't ready to go in F32.
I think the chances of a serious mis-compilation large parts of the distribution
are small. The one mis-compilation we know about was a latent linker bug that
just happened to be triggered by LTO and that particular bug we know how to
identify any packages that might have been broken.
Frankly, there's been more fallout from infrastructure breakage and cmake issues
than anything. I went through the first ~1000 failures proactively looking for
things that were potentially LTO related and fixing them half-dozen or so I
found, but by far the s390 infrastructure and cmake changes have caused more
failures than anything.
As has always been the case, I'm here to address any problems that arise and use
my 30 years of experience with GCC development as well as distribution mass
rebuilds to make informed decisions about the best course of action for any
Yeah, the s390x failures were anoying. I have several ideas to make
things more robust that hopefully we can do before next mass rebuild:
* move the cache host from a z/vm instance to a kvm one.
* We have the kvm ones oversubscribed on cpus, so I'd like to drop all
of them from 4 cpus to 3.
* We might play with the weight on them so koji doesn't run as many jobs
at a time as it does now.
* Make sure ci/koji-simple-ci/koschei isn't doing any long running
builds when the mass rebuild starts. A gcc or libreoffice build can take
up a builder for a long time.
* Run the mass rebuild with --fail-fast so if something fails on some
other arch, it never even needs to run on s390x.
Anyhow, the mass rebuild is over and tagged in. Rawhide compose is
running and should hopefully finish later today.
The second pass took failures from 4162 to 2833, so that helped
a lot: https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html