both mountTests and parted_utils_tests failed.
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log: -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED: <err> = 'losetup: could not find any free loop device\n'; <rc> = 255 --------------------- >> end captured logging << ---------------------
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log: -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED: <err> = 'losetup: could not find any free loop device\n'; <rc> = 255 --------------------- >> end captured logging << ---------------------
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
I googled this error. It seems a Fedora error. It is report that disabling sandbox service and restarting the computer can fix the problem. However the cause is not clear, the bug remains 'new' on the bugzilla. The link is https://bugzilla.redhat.com/show_bug.cgi?id=808795#c31
于 2012年05月23日 03:52, Dan Kenigsberg 写道:
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log: -------------------->> begin captured logging<< -------------------- Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS:<err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n';<rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED:<err> = 'losetup: could not find any free loop device\n';<rc> = 255 --------------------->> end captured logging<< ---------------------
Does your Linux host have a trace of the generated loop device? What says losetup -a ? _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
only change i see from vdsm is this:
Change to use WatchedFileHandler instead of FileHandler in logger.conf (details)
e
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "ShaoHe Feng" shaohef@linux.vnet.ibm.com, eedri@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Adam Litke" agl@us.ibm.com, "Mark Wu" wudxw@linux.vnet.ibm.com, "shu ming" shuming@linux.vnet.ibm.com, "Xu He Jie" xuhj@linux.vnet.ibm.com Sent: Tuesday, May 22, 2012 10:52:41 PM Subject: Re: vdsm unit test error
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log:
-------------------- >> begin captured logging <<
Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED: <err> = 'losetup: could not find any free loop device\n'; <rc> = 255
--------------------- >> end captured logging <<
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
On Wed, May 23, 2012 at 03:50:28AM -0400, Eyal Edri wrote:
only change i see from vdsm is this:
Change to use WatchedFileHandler instead of FileHandler in logger.conf (details)
That's absolutely unrelated. I was refering to whether the Jenkins server was rebooted or fixed in some other means?
e
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "ShaoHe Feng" shaohef@linux.vnet.ibm.com, eedri@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Adam Litke" agl@us.ibm.com, "Mark Wu" wudxw@linux.vnet.ibm.com, "shu ming" shuming@linux.vnet.ibm.com, "Xu He Jie" xuhj@linux.vnet.ibm.com Sent: Tuesday, May 22, 2012 10:52:41 PM Subject: Re: vdsm unit test error
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log:
-------------------- >> begin captured logging <<
Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED: <err> = 'losetup: could not find any free loop device\n'; <rc> = 255
--------------------- >> end captured logging <<
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Eyal Edri" eedri@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Adam Litke" agl@us.ibm.com, "Mark Wu" wudxw@linux.vnet.ibm.com, "shu ming" shuming@linux.vnet.ibm.com, "Xu He Jie" xuhj@linux.vnet.ibm.com, "ShaoHe Feng" shaohef@linux.vnet.ibm.com Sent: Thursday, May 24, 2012 10:42:27 AM Subject: Re: vdsm unit test error
On Wed, May 23, 2012 at 03:50:28AM -0400, Eyal Edri wrote:
only change i see from vdsm is this:
Change to use WatchedFileHandler instead of FileHandler in logger.conf (details)
That's absolutely unrelated. I was refering to whether the Jenkins server was rebooted or fixed in some other means?
not that i know of. there could always be momentary network issues in amazon VMs..
let's keep an eye to see if it returns.
e
----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "ShaoHe Feng" shaohef@linux.vnet.ibm.com, eedri@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Adam Litke" agl@us.ibm.com, "Mark Wu" wudxw@linux.vnet.ibm.com, "shu ming" shuming@linux.vnet.ibm.com, "Xu He Jie" xuhj@linux.vnet.ibm.com Sent: Tuesday, May 22, 2012 10:52:41 PM Subject: Re: vdsm unit test error
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log:
-------------------- >> begin captured logging <<
Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED: <err> = 'losetup: could not find any free loop device\n'; <rc> = 255
--------------------- >> end captured logging <<
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
On 05/23/2012 03:52 AM, Dan Kenigsberg wrote:
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
yes it is reproducible. bu if I shop the sandbox service, and restart, then this problem does not occur. if I sandbox service start, then this problem comes up again.
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log: -------------------->> begin captured logging<< -------------------- Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS:<err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n';<rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED:<err> = 'losetup: could not find any free loop device\n';<rc> = 255 --------------------->> end captured logging<< ---------------------
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
# losetup -a /dev/loop0: [fd03]:918121 (/tmp/tmpeihztM) /dev/loop1: [fd03]:918122 (/tmp/tmp43EVnb) /dev/loop2: [fd03]:918123 (/tmp/tmpCoknYi) /dev/loop3: [fd03]:918124 (/tmp/tmp_PFqBx) /dev/loop4: [fd03]:918125 (/tmp/tmplVEPQs) /dev/loop5: [fd03]:918126 (/tmp/tmpQrHVKH) /dev/loop6: [fd03]:918127 (/tmp/tmpZkZJ7V) /dev/loop7: [fd03]:918128 (/tmp/tmpmUSR26)
# losetup -d /dev/loop0 loop: can't delete device /dev/loop0: Device or resource busy
# lsof -L | grep loop loop0 11198 root cwd DIR 253,3 4096 2 / loop0 11198 root rtd DIR 253,3 4096 2 / loop0 11198 root txt unknown /proc/11198/exe loop1 11287 root cwd DIR 253,3 4096 2 / loop1 11287 root rtd DIR 253,3 4096 2 / loop1 11287 root txt unknown /proc/11287/exe loop2 11309 root cwd DIR 253,3 4096 2 / loop2 11309 root rtd DIR 253,3 4096 2 / loop2 11309 root txt unknown /proc/11309/exe loop3 11327 root cwd DIR 253,3 4096 2 / loop3 11327 root rtd DIR 253,3 4096 2 / loop3 11327 root txt unknown /proc/11327/exe loop4 11350 root cwd DIR 253,3 4096 2 / loop4 11350 root rtd DIR 253,3 4096 2 / loop4 11350 root txt unknown /proc/11350/exe loop5 11372 root cwd DIR 253,3 4096 2 / loop5 11372 root rtd DIR 253,3 4096 2 / loop5 11372 root txt unknown /proc/11372/exe loop6 11391 root cwd DIR 253,3 4096 2 / loop6 11391 root rtd DIR 253,3 4096 2 / loop6 11391 root txt unknown /proc/11391/exe loop7 11408 root cwd DIR 253,3 4096 2 / loop7 11408 root rtd DIR 253,3 4096 2 / loop7 11408 root txt unknown /proc/11408/exe
should I use the strace to watch the syscall about what happened to /dev/loop # strace -f -F -o ./strace.log ./run_tests.sh mountTests or any other way to get more info?
On Fri, May 25, 2012 at 02:59:31AM +0800, ShaoHe Feng wrote:
On 05/23/2012 03:52 AM, Dan Kenigsberg wrote:
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
yes it is reproducible. bu if I shop the sandbox service, and restart, then this problem does not occur. if I sandbox service start, then this problem comes up again.
Pardon, but I am not familiar with the sandbox service. Could you describe a complete reproducer?
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log: -------------------->> begin captured logging<< -------------------- Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS:<err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n';<rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED:<err> = 'losetup: could not find any free loop device\n';<rc> = 255 --------------------->> end captured logging<< ---------------------
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
# losetup -a /dev/loop0: [fd03]:918121 (/tmp/tmpeihztM) /dev/loop1: [fd03]:918122 (/tmp/tmp43EVnb) /dev/loop2: [fd03]:918123 (/tmp/tmpCoknYi) /dev/loop3: [fd03]:918124 (/tmp/tmp_PFqBx) /dev/loop4: [fd03]:918125 (/tmp/tmplVEPQs) /dev/loop5: [fd03]:918126 (/tmp/tmpQrHVKH) /dev/loop6: [fd03]:918127 (/tmp/tmpZkZJ7V) /dev/loop7: [fd03]:918128 (/tmp/tmpmUSR26)
# losetup -d /dev/loop0 loop: can't delete device /dev/loop0: Device or resource busy
# lsof -L | grep loop loop0 11198 root cwd DIR 253,3 4096 2 /
Who is process 11198 ?
loop0 11198 root rtd DIR 253,3 4096 2 / loop0 11198 root txt unknown /proc/11198/exe loop1 11287 root cwd DIR 253,3 4096 2 / loop1 11287 root rtd DIR 253,3 4096 2 / loop1 11287 root txt unknown /proc/11287/exe loop2 11309 root cwd DIR 253,3 4096 2 / loop2 11309 root rtd DIR 253,3 4096 2 / loop2 11309 root txt unknown /proc/11309/exe loop3 11327 root cwd DIR 253,3 4096 2 / loop3 11327 root rtd DIR 253,3 4096 2 / loop3 11327 root txt unknown /proc/11327/exe loop4 11350 root cwd DIR 253,3 4096 2 / loop4 11350 root rtd DIR 253,3 4096 2 / loop4 11350 root txt unknown /proc/11350/exe loop5 11372 root cwd DIR 253,3 4096 2 / loop5 11372 root rtd DIR 253,3 4096 2 / loop5 11372 root txt unknown /proc/11372/exe loop6 11391 root cwd DIR 253,3 4096 2 / loop6 11391 root rtd DIR 253,3 4096 2 / loop6 11391 root txt unknown /proc/11391/exe loop7 11408 root cwd DIR 253,3 4096 2 / loop7 11408 root rtd DIR 253,3 4096 2 / loop7 11408 root txt unknown /proc/11408/exe
should I use the strace to watch the syscall about what happened to /dev/loop # strace -f -F -o ./strace.log ./run_tests.sh mountTests or any other way to get more info?
On 05/25/2012 04:52 PM, Dan Kenigsberg wrote:
On Fri, May 25, 2012 at 02:59:31AM +0800, ShaoHe Feng wrote:
On 05/23/2012 03:52 AM, Dan Kenigsberg wrote:
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
yes it is reproducible. bu if I shop the sandbox service, and restart, then this problem does not occur. if I sandbox service start, then this problem comes up again.
Pardon, but I am not familiar with the sandbox service. Could you describe a complete reproducer?
# chkconfig sandbox off then restart fedora. # ./run_tests.sh mountTests
no matter how times the test rum, then this problem does not comes up.
# chkconfig sandbox on then restart fedora. # ./run_tests.sh mountTests # ls /dev/loop* there are 7 loop devices. and I run the test 8 times, then the problem comes up.
losetup: could not find any free loop device
however after I make one loop block file by mknod , I can run test successfully one time.
We, or some other suite using the server, may be leaking a loop device. Eyal Edri, do you know what has made this go away in run #144?
the test execute 'mount' and 'umount' command. after the umount command, the loop device can not be freed
here is the log: -------------------->> begin captured logging<< -------------------- Storage.Misc.excCmd: DEBUG: 'dd if=/dev/zero of=/tmp/tmpH2KSCr bs=100M count=1' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS:<err> = '1+0 records in\n1+0 records out\n104857600 bytes (105 MB) copied, 0.266024 s, 39 4 MB/s\n';<rc> = 0 Storage.Misc.excCmd: DEBUG: 'losetup -f --show /tmp/tmpH2KSCr' (cwd None) Storage.Misc.excCmd: DEBUG: FAILED:<err> = 'losetup: could not find any free loop device\n';<rc> = 255 --------------------->> end captured logging<< ---------------------
Does your Linux host have a trace of the generated loop device? What says losetup -a ?
# losetup -a /dev/loop0: [fd03]:918121 (/tmp/tmpeihztM) /dev/loop1: [fd03]:918122 (/tmp/tmp43EVnb) /dev/loop2: [fd03]:918123 (/tmp/tmpCoknYi) /dev/loop3: [fd03]:918124 (/tmp/tmp_PFqBx) /dev/loop4: [fd03]:918125 (/tmp/tmplVEPQs) /dev/loop5: [fd03]:918126 (/tmp/tmpQrHVKH) /dev/loop6: [fd03]:918127 (/tmp/tmpZkZJ7V) /dev/loop7: [fd03]:918128 (/tmp/tmpmUSR26)
# losetup -d /dev/loop0 loop: can't delete device /dev/loop0: Device or resource busy
# lsof -L | grep loop loop0 11198 root cwd DIR 253,3 4096 2 /
Who is process 11198 ?
it is a kernel thread. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 11198 0.0 0.0 0 0 ? S< 02:32 0:00 [loop0]
loop0 11198 root rtd DIR 253,3 4096 2 / loop0 11198 root txt unknown /proc/11198/exe loop1 11287 root cwd DIR 253,3 4096 2 / loop1 11287 root rtd DIR 253,3 4096 2 / loop1 11287 root txt unknown /proc/11287/exe loop2 11309 root cwd DIR 253,3 4096 2 / loop2 11309 root rtd DIR 253,3 4096 2 / loop2 11309 root txt unknown /proc/11309/exe loop3 11327 root cwd DIR 253,3 4096 2 / loop3 11327 root rtd DIR 253,3 4096 2 / loop3 11327 root txt unknown /proc/11327/exe loop4 11350 root cwd DIR 253,3 4096 2 / loop4 11350 root rtd DIR 253,3 4096 2 / loop4 11350 root txt unknown /proc/11350/exe loop5 11372 root cwd DIR 253,3 4096 2 / loop5 11372 root rtd DIR 253,3 4096 2 / loop5 11372 root txt unknown /proc/11372/exe loop6 11391 root cwd DIR 253,3 4096 2 / loop6 11391 root rtd DIR 253,3 4096 2 / loop6 11391 root txt unknown /proc/11391/exe loop7 11408 root cwd DIR 253,3 4096 2 / loop7 11408 root rtd DIR 253,3 4096 2 / loop7 11408 root txt unknown /proc/11408/exe
should I use the strace to watch the syscall about what happened to /dev/loop # strace -f -F -o ./strace.log ./run_tests.sh mountTests or any other way to get more info?
On Fri, May 25, 2012 at 05:40:15PM +0800, ShaoHe Feng wrote:
On 05/25/2012 04:52 PM, Dan Kenigsberg wrote:
On Fri, May 25, 2012 at 02:59:31AM +0800, ShaoHe Feng wrote:
On 05/23/2012 03:52 AM, Dan Kenigsberg wrote:
On Tue, May 22, 2012 at 05:02:01PM +0800, ShaoHe Feng wrote:
both mountTests and parted_utils_tests failed.
Failed where? On your own host? Is it reproducible? We had a similar, but transient, problem in http://jenkins.ovirt.org/job/vdsm_unit_tests/143/console
yes it is reproducible. bu if I shop the sandbox service, and restart, then this problem does not occur. if I sandbox service start, then this problem comes up again.
Pardon, but I am not familiar with the sandbox service. Could you describe a complete reproducer?
# chkconfig sandbox off then restart fedora. # ./run_tests.sh mountTests
no matter how times the test rum, then this problem does not comes up.
# chkconfig sandbox on then restart fedora. # ./run_tests.sh mountTests # ls /dev/loop* there are 7 loop devices. and I run the test 8 times, then the problem comes up.
losetup: could not find any free loop device
however after I make one loop block file by mknod , I can run test successfully one time.
Since on another thread you suspected that this is a kernel issue, could you state for the record your kernel version, as well as policycoreutils-sandbox's ?
Maybe this could give a hint for someone.
Thanks!
vdsm-devel@lists.fedorahosted.org