On Sat, Aug 01, 2020 at 02:03:40PM -0600, Jeff Law wrote:
> On Sat, 2020-08-01 at 12:12 +0200, Kevin Kofler wrote:
> > Hi,
> > seeing the amount of fallout from LTO, I really
think that this feature
> > ought to be dropped from F33, and evaluated carefully for F34 (i.e., can it
> > be done without breaking the build of or miscompiling a large part of the
> > distribution, once the bugs such as the ld bug discussed in this thread are
> > fixed, or is it just unsafe to enable by default to begin with?). I.e.,
> > revert it for F33 for sure, then decide whether to retry it for F34 or can
> > it permanently.
> Most of the fallout has been Nick pushing through binutils builds that are
> broken. Seriously, there's been at least 4 builds pushed through that kept
> bringing back the *same* problem.
> And just to be clear, this has been 6+ months of behind
the scenes work to find
> and identify issues, fix broken packages, put global mitigations of broken crap
> in place in place, opt-out packages that do things that are fundamentally
> incompatible with LTO, etc. In fact it was that behind-the-scenes work that
> pushed this feature from F32 to F33 as it just wasn't ready to go in F32.
> I think the chances of a serious mis-compilation large
parts of the distribution
> are small. The one mis-compilation we know about was a latent linker bug that
> just happened to be triggered by LTO and that particular bug we know how to
> identify any packages that might have been broken.
> Frankly, there's been more fallout from
infrastructure breakage and cmake issues
> than anything. I went through the first ~1000 failures proactively looking for
> things that were potentially LTO related and fixing them half-dozen or so I
> found, but by far the s390 infrastructure and cmake changes have caused more
> failures than anything.
> As has always been the case, I'm here to address any
problems that arise and use
> my 30 years of experience with GCC development as well as distribution mass
> rebuilds to make informed decisions about the best course of action for any
> particular problem.
Yeah, the s390x failures were anoying. I have several ideas to make
things more robust that hopefully we can do before next mass rebuild:
* move the cache host from a z/vm instance to a kvm one.
* We have the kvm ones oversubscribed on cpus, so I'd like to drop all
of them from 4 cpus to 3.
* We might play with the weight on them so koji doesn't run as many jobs
at a time as it does now.
* Make sure ci/koji-simple-ci/koschei isn't doing any long running
builds when the mass rebuild starts. A gcc or libreoffice build can take
up a builder for a long time.
* Run the mass rebuild with --fail-fast so if something fails on some
other arch, it never even needs to run on s390x.
Anyhow, the mass rebuild is over and tagged in. Rawhide compose is
running and should hopefully finish later today.
The second pass took failures from 4162 to 2833, so that helped
a lot: https://kojipkgs.fedoraproject.org/mass-rebuild/f33-failures.html
Thanks for the update. I'll start scanning through those. It looks like
some of the cmake things are getting addressed which I'm sure helps too.