Hi!
I've been trying to build gcc in rawhide and f35 for more than a week now, but haven't succeeded due to instability of the armv7hl builders. E.g. the https://koji.fedoraproject.org/koji/taskinfo?taskID=77117620 build succeeded on all but armv7hl in less than 22 hours (on aarch64 even took less than 5 hours), but armv7hl kept rebooting or whatever (impossible to find out from the logs) 8 times, see https://koji.fedoraproject.org/koji/taskinfo?taskID=77117797 Buildroots list. Eventually I've cancelled that build after 151 hours, disabled LTO on armv7hl (for GCC bootstrap) and am building now https://koji.fedoraproject.org/koji/taskinfo?taskID=77518894 https://koji.fedoraproject.org/koji/taskinfo?taskID=77518968 but those again show total time of ~ 21 hours, but armv7hl task times of 1.5 and 10 hours, so the f35 build was retried already 2 times and f36 once.
Any recent kernel changes on the boxes, or is there any way to find out what's going on with them, whether it oopses, OOMs or something else?
In late August the package built just fine and took ~ 27 hours to build on armv7hl.
Jakub
I asked this in another thread, but maybe this is related to [1]? How do we disable just link-time parallelism?
[1] https://src.fedoraproject.org/rpms/redhat-rpm-config/c/bc8fa85e907d4b2b88760...
On Wed, 20 Oct 2021 at 15:15, Jakub Jelinek jakub@redhat.com wrote:
Hi!
I've been trying to build gcc in rawhide and f35 for more than a week now, but haven't succeeded due to instability of the armv7hl builders. E.g. the https://koji.fedoraproject.org/koji/taskinfo?taskID=77117620 build succeeded on all but armv7hl in less than 22 hours (on aarch64 even took less than 5 hours), but armv7hl kept rebooting or whatever (impossible to find out from the logs) 8 times, see https://koji.fedoraproject.org/koji/taskinfo?taskID=77117797 Buildroots list. Eventually I've cancelled that build after 151 hours, disabled LTO on armv7hl (for GCC bootstrap) and am building now https://koji.fedoraproject.org/koji/taskinfo?taskID=77518894 https://koji.fedoraproject.org/koji/taskinfo?taskID=77518968 but those again show total time of ~ 21 hours, but armv7hl task times of 1.5 and 10 hours, so the f35 build was retried already 2 times and f36 once.
Any recent kernel changes on the boxes, or is there any way to find out what's going on with them, whether it oopses, OOMs or something else?
In late August the package built just fine and took ~ 27 hours to build on armv7hl.
Jakub
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, 2021-10-20 at 15:22 +0200, Iñaki Ucar wrote: I asked this in another thread, but maybe this is related to [1]? How do we disable just link-time parallelism?
[1] https://src.fedoraproject.org/rpms/redhat-rpm-config/c/bc8fa85e907d4b2b88760...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On Wed, 20 Oct 2021 at 15:15, Jakub Jelinek jakub@redhat.com wrote:
Hi!
I've been trying to build gcc in rawhide and f35 for more than a week now, but haven't succeeded due to instability of the armv7hl builders. E.g. the https://koji.fedoraproject.org/koji/taskinfo?taskID=77117620 build succeeded on all but armv7hl in less than 22 hours (on aarch64 even took less than 5 hours), but armv7hl kept rebooting or whatever (impossible to find out from the logs) 8 times, see https://koji.fedoraproject.org/koji/taskinfo?taskID=77117797 Buildroots list. Eventually I've cancelled that build after 151 hours, disabled LTO on armv7hl (for GCC bootstrap) and am building now https://koji.fedoraproject.org/koji/taskinfo?taskID=77518894 https://koji.fedoraproject.org/koji/taskinfo?taskID=77518968 but those again show total time of ~ 21 hours, but armv7hl task times of 1.5 and 10 hours, so the f35 build was retried already 2 times and f36 once.
Any recent kernel changes on the boxes, or is there any way to find out what's going on with them, whether it oopses, OOMs or something else?
In late August the package built just fine and took ~ 27 hours to build on armv7hl.
Jakub _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, Oct 20, 2021 at 03:22:13PM +0200, Iñaki Ucar wrote:
I asked this in another thread, but maybe this is related to [1]? How do we disable just link-time parallelism?
gcc uses its --with-build-config=bootstrap-lto --enable-link-serialization=1 when building gcc with LTO is enabled and the latter option ensures through Makefile dependencies that for the large link commands only one is done at a time. And, for the new builds I've disabled the building with LTO, i.e. %if 0%{?fedora} >= 35 || 0%{?rhel} >= 9 %ifnarch %{arm} --with-build-config=bootstrap-lto --enable-link-serialization=1 \ %endif %endif and so it should consume even less memory. Also, even when watching the LTO build that took more than 6 days, several times I saw it going into the %check phase with running just the testsuite.
Jakub
On Wed, Oct 20, 2021 at 03:06:21PM +0200, Jakub Jelinek wrote:
Hi!
I've been trying to build gcc in rawhide and f35 for more than a week now, but haven't succeeded due to instability of the armv7hl builders. E.g. the https://koji.fedoraproject.org/koji/taskinfo?taskID=77117620 build succeeded on all but armv7hl in less than 22 hours (on aarch64 even took less than 5 hours), but armv7hl kept rebooting or whatever (impossible to find out from the logs) 8 times, see https://koji.fedoraproject.org/koji/taskinfo?taskID=77117797 Buildroots list.
It's kojid being OOM killed. So, it restarts, sees that it's not running the build it's supposed to be, starts it over, gets OOM killed, repeat.
Eventually I've cancelled that build after 151 hours, disabled LTO on armv7hl (for GCC bootstrap) and am building now https://koji.fedoraproject.org/koji/taskinfo?taskID=77518894 https://koji.fedoraproject.org/koji/taskinfo?taskID=77518968 but those again show total time of ~ 21 hours, but armv7hl task times of 1.5 and 10 hours, so the f35 build was retried already 2 times and f36 once.
Any recent kernel changes on the boxes, or is there any way to find out what's going on with them, whether it oopses, OOMs or something else?
Its OOM.
It's basically this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1920183 It's much less common with the kernels we are running ( 5.12.19-300.fc34.armv7hl+lpae ) but it still does hit some packages. :(
Nothing has changed on the builders. We are in final freeze and the only changes made on them are security updates, and none of those look like they would affect builds (vim, libjpeg-turbo, libvirt, openssh, curl)
I can try f35 + latest kernel in stg and see if it's any better or this bug magically got fixed. :(
In late August the package built just fine and took ~ 27 hours to build on armv7hl.
Things should be like they were in late Aug. :(
kevin