Hello,
I am trying to build new bind on Rawhide, but I suddenly get quite high rate of failures during builds. Strange is the build is always successful, when I try fedpkg mockbuild on my machine.
I tried several tries [1][2][3] and scratch builds fail often. Even production builds fail, but I think they haven't failed until recently. I were unable to reproduce it at least once in mockbuild.
Is it possible to record crashes during builds using abrt or faf? Is there way to obtain more details of failed builds? Is there anything similar to mockbuild --no-cleanup-after, which could help me identify the cause?
Do you have any hints? I tried deleting /var/lib/mock and /var/cache/mock, but that haven't changed anything. COPR [4] haven't failed also.
Regards, Petr
1. https://koji.fedoraproject.org/koji/taskinfo?taskID=59974310 2. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985548 3. http://koji.fedoraproject.org/koji/taskinfo?taskID=59985586 4. https://copr.fedorainfracloud.org/coprs/pemensik/bind-9.16/build/1886936/
On Mon, Jan 18, 2021 at 11:22:10PM +0100, Petr Menšík wrote:
Hello,
I am trying to build new bind on Rawhide, but I suddenly get quite high rate of failures during builds. Strange is the build is always successful, when I try fedpkg mockbuild on my machine.
I tried several tries [1][2][3] and scratch builds fail often. Even production builds fail, but I think they haven't failed until recently. I were unable to reproduce it at least once in mockbuild.
Is it possible to record crashes during builds using abrt or faf? Is there way to obtain more details of failed builds? Is there anything similar to mockbuild --no-cleanup-after, which could help me identify the cause?
Do you have any hints? I tried deleting /var/lib/mock and /var/cache/mock, but that haven't changed anything. COPR [4] haven't failed also.
I note that all of these seem to have failed on buildhw-x86 builders. Those are bare metal hardware with tons of cpu and memory. Could be something particular to them?
Otherwise it looks like a toolchain issue perhaps?
You could try adding more debugging to the test suite and doing scratch builds? Perhaps the tools folks will have some subbestion...
kevin
It is tedious, but you can try gdb in batch mode, e.g.:
~~~ gdb /usr/bin/ruby-mri -batch \ -ex 'set breakpoint pending on' \ -ex 'r -Ilib:test -e '''Dir.glob %|./test/**/*_test.rb|, &method(:require)''' -- -v' \ -ex 'bt full' \ -ex c \ -ex 'bt full' \ -ex quit ~~~
(and of course modify this to fit your use case).
Vít
Dne 18. 01. 21 v 23:22 Petr Menšík napsal(a):
Hello,
I am trying to build new bind on Rawhide, but I suddenly get quite high rate of failures during builds. Strange is the build is always successful, when I try fedpkg mockbuild on my machine.
I tried several tries [1][2][3] and scratch builds fail often. Even production builds fail, but I think they haven't failed until recently. I were unable to reproduce it at least once in mockbuild.
Is it possible to record crashes during builds using abrt or faf? Is there way to obtain more details of failed builds? Is there anything similar to mockbuild --no-cleanup-after, which could help me identify the cause?
Do you have any hints? I tried deleting /var/lib/mock and /var/cache/mock, but that haven't changed anything. COPR [4] haven't failed also.
Regards, Petr
- https://koji.fedoraproject.org/koji/taskinfo?taskID=59974310
- http://koji.fedoraproject.org/koji/taskinfo?taskID=59985548
- http://koji.fedoraproject.org/koji/taskinfo?taskID=59985586
- https://copr.fedorainfracloud.org/coprs/pemensik/bind-9.16/build/1886936/
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Thanks for the tips. I eventually narrowed errors to just NUMA builders, and confirmed that once borrowed multicore machine from RH.
Turned out the problem is internal to BIND only, reported under MR [1]. It is problematic to hide details on kyua unit testing. Unit tests are built using cmocka and can print nice output. Unfortunately it uses kyua to group multiple results and that tool hides any details. Without easy way to provide them. Not sure which tool should be updated, kyua is the best candidate for more detailed output.
It should be always easy to make building more verbose, especially in such weird errors.
Thanks for help.
1. https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/4582
On 1/19/21 5:32 PM, Vít Ondruch wrote:
It is tedious, but you can try gdb in batch mode, e.g.:
gdb /usr/bin/ruby-mri -batch \ -ex 'set breakpoint pending on' \ -ex 'r -Ilib:test -e '\''Dir.glob %|./test/**/*_test.rb|, &method(:require)'\'' -- -v' \ -ex 'bt full' \ -ex c \ -ex 'bt full' \ -ex quit
(and of course modify this to fit your use case).
Vít
Dne 18. 01. 21 v 23:22 Petr Menšík napsal(a):
Hello,
I am trying to build new bind on Rawhide, but I suddenly get quite high rate of failures during builds. Strange is the build is always successful, when I try fedpkg mockbuild on my machine.
I tried several tries [1][2][3] and scratch builds fail often. Even production builds fail, but I think they haven't failed until recently. I were unable to reproduce it at least once in mockbuild.
Is it possible to record crashes during builds using abrt or faf? Is there way to obtain more details of failed builds? Is there anything similar to mockbuild --no-cleanup-after, which could help me identify the cause?
Do you have any hints? I tried deleting /var/lib/mock and /var/cache/mock, but that haven't changed anything. COPR [4] haven't failed also.
Regards, Petr
- https://koji.fedoraproject.org/koji/taskinfo?taskID=59974310
- http://koji.fedoraproject.org/koji/taskinfo?taskID=59985548
- http://koji.fedoraproject.org/koji/taskinfo?taskID=59985586
https://copr.fedorainfracloud.org/coprs/pemensik/bind-9.16/build/1886936/
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org