Hello, something with the build system is wrong. The build self will work, but the system says failed. Have anyone see this also?
Build ID: 36210 and 36211.
Thanks.
On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
Hello, something with the build system is wrong. The build self will work, but the system says failed. Have anyone see this also?
Build ID: 36210 and 36211.
It has been a topic on fedora-maintainer multiple times. Here's the corresponding ticket:
https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
Michael Schwendt schrieb:
On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
Hello, something with the build system is wrong. The build self will work, but the system says failed. Have anyone see this also?
Build ID: 36210 and 36211.
It has been a topic on fedora-maintainer multiple times. Here's the corresponding ticket:
https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
Yes, it's my problem too. I hope it will be fixed soon.
On Sat, 08 Sep 2007 11:01:24 +0200, Frank Büttner wrote:
Michael Schwendt schrieb:
On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
Hello, something with the build system is wrong. The build self will work, but the system says failed. Have anyone see this also?
Build ID: 36210 and 36211.
It has been a topic on fedora-maintainer multiple times. Here's the corresponding ticket:
https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
Yes, it's my problem too. I hope it will be fixed soon.
Consider this fixed [1]. Resubmit your failed build-jobs for Extras and EPEL and see.
After I'd noticed that approx. 5 out of 6 noarch build-jobs on the ppc2 server failed with the same symptoms, I've picked yum-cron as a test pkg for an online-debugging session on the ppc2 host. As another pkg that had failed very often, I've tested the final patch with ctapi-common.
The reason for failure was a race condition between plague 0.4.4.1 and mock. The plague builder did not monitor the mock state accurately enough. Mock could change its state (e.g. from prepping/setup to building) while the plague builder was sleeping. In the cases I've observed, mock had exited already when plague woke up and made a wrong assumption about the state mock was in when it exited.
[1] http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch
On Wed, 19 Sep 2007 03:59:38 +0200 Michael Schwendt mschwendt.tmp0701.nospam@arcor.de wrote:
The reason for failure was a race condition between plague 0.4.4.1 and mock. The plague builder did not monitor the mock state accurately enough. Mock could change its state (e.g. from prepping/setup to building) while the plague builder was sleeping. In the cases I've observed, mock had exited already when plague woke up and made a wrong assumption about the state mock was in when it exited.
[1] http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch
This is really awesome that you were able to track this down. Thank you very much for this work.
On Wed, 2007-09-19 at 03:59 +0200, Michael Schwendt wrote:
On Sat, 08 Sep 2007 11:01:24 +0200, Frank Büttner wrote:
Michael Schwendt schrieb:
On Sat, 08 Sep 2007 09:36:21 +0200, Frank Büttner wrote:
Hello, something with the build system is wrong. The build self will work, but the system says failed. Have anyone see this also?
Build ID: 36210 and 36211.
It has been a topic on fedora-maintainer multiple times. Here's the corresponding ticket:
https://hosted.fedoraproject.org/projects/fedora-infrastructure/ticket/125
Yes, it's my problem too. I hope it will be fixed soon.
Consider this fixed [1]. Resubmit your failed build-jobs for Extras and EPEL and see.
After I'd noticed that approx. 5 out of 6 noarch build-jobs on the ppc2 server failed with the same symptoms, I've picked yum-cron as a test pkg for an online-debugging session on the ppc2 host. As another pkg that had failed very often, I've tested the final patch with ctapi-common.
The reason for failure was a race condition between plague 0.4.4.1 and mock. The plague builder did not monitor the mock state accurately enough. Mock could change its state (e.g. from prepping/setup to building) while the plague builder was sleeping. In the cases I've observed, mock had exited already when plague woke up and made a wrong assumption about the state mock was in when it exited.
[1] http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch
Workaround looks sane, diagnosis is correct, thanks for tracking this down and fixing it. Out of curiosity, did a mock update or a hardware change exacerbate the issue? We only just started seeing this after ~2 years of using plague.
Dan
Once upon a time Tuesday 18 September 2007, Dan Williams wrote:
Workaround looks sane, diagnosis is correct, thanks for tracking this down and fixing it. Out of curiosity, did a mock update or a hardware change exacerbate the issue? We only just started seeing this after ~2 years of using plague.
Dan
not really sure, mock has not been updated on the builders for awhile. there has been no hardware changes. the only real difference that i see is that plague is being used less now. we are building EPEL-4 EPEL-5 and Extras 6 in plague. currently we have been keeping 4 cores available for x86_64 and 4 for ppc.
Dennis
On Tue, 18 Sep 2007 22:36:15 -0400, Dan Williams wrote:
[1] http://home.arcor.de/ms2002sep/tmp/plague-0.4.4.1-racecondition.patch
Workaround looks sane, diagnosis is correct, thanks for tracking this down and fixing it. Out of curiosity, did a mock update or a hardware change exacerbate the issue? We only just started seeing this after ~2 years of using plague.
I have doubts that the .../mock-state/status file is in sync with mock. It either shows symptoms of buffering or changes too quickly. It stays in "prep" for most of the time, then very quickly steps through "setu" -> "buil" -> "done". So quickly that plague-builder cannot sleep for 3 secs without missing the state changes before mock is done.
Perhaps this is different for longer builds which don't run for just 6-8 minutes.
Look at this debug log excerpt made when plague-builder was monitoring the "prepping" step (the "string" value is the four chars from the mock-state file after a refresh):
Running _status_prepping: prepping string = prep _watch_mock(): waitpid = 0,0 continuing _status_prepping: prepping string = prep Running _status_prepping: prepping string = done
^ here mock-state changed while plague-builder was sleeping for 3s => from "prep" to mock-exit in plague-builder's view (!) => when not reloading mock's state file to verify
string = done _watch_mock(): waitpid = 4139,0 _watch_mock(): PID 4139 exit status 0 Setting Status Good! - failed
^ this is based on the assumption that a mock-exit during "prepping" would always be an error condition despite the zero exit code
init clean prep This may take a while setup build ending done
^ this is what mock has processed actually while plague-builder assumed that mock still was doing only "prep"+"setu"
Results and/or logs in: /mnt/build/builder_work/46bb3113e41a4745e4b1217cb9d5b6b8ef5cc73b/result Cleaning up the buildroot... /usr/bin/setarch ppc32 /usr/bin/mock clean --uniqueext=46bb3113e41a4745e4b1217cb9d5b6b8ef5cc73b -r fedora-6-ppc-core