Sometimes we get this error when running the vdsm tests:
====================================================================== ERROR: testLoopMount (mountTests.MountTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line 69, in testLoopMount m.mount(mntOpts="loop") File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 222, in mount return self._runcmd(cmd, timeout) File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 238, in _runcmd raise MountError(rc, ";".join((out, err))) MountError: (2, ';mount: could not find any free loop device\n') -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 (17-May-2010)\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None)
The problem is that it seems that the loop devices that are not being released (maybe when the test fails?) and the system runs out of devices eventually. Can you take a look to see where the cleanup fails and fix it?
Thanks!
pd. To free a loop device you have to umount it and then losetup -d <loopdevice>, that way the device gets liberated. Also it might be a good solution to create a specific device per test, that way it will never run out of them no matter how many tests run in parallel, but that requires a little more modifications.
---- David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
On Fri 09 Aug 2013 09:45:02 PM CST, David Caro Estevez wrote:
Sometimes we get this error when running the vdsm tests:
====================================================================== ERROR: testLoopMount (mountTests.MountTests)
Traceback (most recent call last): File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line 69, in testLoopMount m.mount(mntOpts="loop") File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 222, in mount return self._runcmd(cmd, timeout) File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 238, in _runcmd raise MountError(rc, ";".join((out, err))) MountError: (2, ';mount: could not find any free loop device\n') -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 (17-May-2010)\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None)
The problem is that it seems that the loop devices that are not being released (maybe when the test fails?) and the system runs out of devices eventually. Can you take a look to see where the cleanup fails and fix it?
It seems the system run out of loop devices. You could use 'losetup -l' to check the usages of the loop devices. If there's some leaks, you could use losetup to fix it manually. Otherwise, that means you need increase the number of loop devices. How to do that depends on your distro:
On latest fedora, loop module is kernel built-in, so you have to change the boot kernel parameter: append to the end of each linux line in /boot/grub2/grub.cfg: max_loop=64
For distros which loop modules is loaded as a separate module: enter following line in /etc/modprobe.conf options loop max_loop=64
Hope it helps.
Thanks!
pd. To free a loop device you have to umount it and then losetup -d <loopdevice>, that way the device gets liberated. Also it might be a good solution to create a specific device per test, that way it will never run out of them no matter how many tests run in parallel, but that requires a little more modifications.
David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Hi David,
on 2013-08-09 21:45, David Caro Estevez wrote:
Sometimes we get this error when running the vdsm tests:
====================================================================== ERROR: testLoopMount (mountTests.MountTests)
Traceback (most recent call last): File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line 69, in testLoopMount m.mount(mntOpts="loop") File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 222, in mount return self._runcmd(cmd, timeout) File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 238, in _runcmd raise MountError(rc, ";".join((out, err))) MountError: (2, ';mount: could not find any free loop device\n') -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 (17-May-2010)\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None)
The problem is that it seems that the loop devices that are not being released (maybe when the test fails?) and the system runs out of devices eventually. Can you take a look to see where the cleanup fails and fix it?
I think this may be related to a known bug [1] of gvfs.
To workaround the bug, 1. Reboot Linux 2. See if "ps aux | grep gvfs" produces any results, then kill all gvfs related processes. 3. "cd tests", and run the tests "./run_tests_local.sh mkimageTests.py mountTests.py" for 10 times, and "losetup -a" should give empty result after each run. When gvfs is running, "losetup -a" would show 2 new loop devices are occupied after each run.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=853674#c5 Thanks and best regards!
Zhou Zheng Sheng / 周征晟 E-mail: zhshzhou@linux.vnet.ibm.com Telephone: 86-10-82454397
On Mon 12 Aug 2013 04:13:26 AM CEST, Zhou Zheng Sheng wrote:
Hi David,
on 2013-08-09 21:45, David Caro Estevez wrote:
Sometimes we get this error when running the vdsm tests:
====================================================================== ERROR: testLoopMount (mountTests.MountTests)
Traceback (most recent call last): File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line 69, in testLoopMount m.mount(mntOpts="loop") File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 222, in mount return self._runcmd(cmd, timeout) File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 238, in _runcmd raise MountError(rc, ";".join((out, err))) MountError: (2, ';mount: could not find any free loop device\n') -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 (17-May-2010)\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None)
The problem is that it seems that the loop devices that are not being released (maybe when the test fails?) and the system runs out of devices eventually. Can you take a look to see where the cleanup fails and fix it?
I think this may be related to a known bug [1] of gvfs.
To workaround the bug,
- Reboot Linux
- See if "ps aux | grep gvfs" produces any results, then kill all gvfs
related processes. 3. "cd tests", and run the tests "./run_tests_local.sh mkimageTests.py mountTests.py" for 10 times, and "losetup -a" should give empty result after each run. When gvfs is running, "losetup -a" would show 2 new loop devices are occupied after each run.
gvfs is not installed on the machines, so I doubt this is the same error :( I suggest to add some cleanup code at the end of the tests to free all the used loop devices (if you free only the ones you use, it will not fail when running parallel tests).
Adding more loop devices or cleaning them all manually is only a temporary solution.
Thanks!
-- David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
Hi David,
on 2013-08-12 14:59, David Caro wrote:
On Mon 12 Aug 2013 04:13:26 AM CEST, Zhou Zheng Sheng wrote:
Hi David,
on 2013-08-09 21:45, David Caro Estevez wrote:
Sometimes we get this error when running the vdsm tests:
====================================================================== ERROR: testLoopMount (mountTests.MountTests)
Traceback (most recent call last): File "/ephemeral0/vdsm_unit_tests_gerrit_el/tests/mountTests.py", line 69, in testLoopMount m.mount(mntOpts="loop") File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 222, in mount return self._runcmd(cmd, timeout) File "/ephemeral0/vdsm_unit_tests_gerrit_el/vdsm/storage/mount.py", line 238, in _runcmd raise MountError(rc, ";".join((out, err))) MountError: (2, ';mount: could not find any free loop device\n') -------------------- >> begin captured logging << -------------------- Storage.Misc.excCmd: DEBUG: '/sbin/mkfs.ext2 -F /tmp/tmpq95svr' (cwd None) Storage.Misc.excCmd: DEBUG: SUCCESS: <err> = 'mke2fs 1.41.12 (17-May-2010)\n'; <rc> = 0 Storage.Misc.excCmd: DEBUG: '/usr/bin/sudo -n /bin/mount -o loop /tmp/tmpq95svr /tmp/tmpcS29EU' (cwd None)
The problem is that it seems that the loop devices that are not being released (maybe when the test fails?) and the system runs out of devices eventually. Can you take a look to see where the cleanup fails and fix it?
I think this may be related to a known bug [1] of gvfs.
To workaround the bug,
- Reboot Linux
- See if "ps aux | grep gvfs" produces any results, then kill all gvfs
related processes. 3. "cd tests", and run the tests "./run_tests_local.sh mkimageTests.py mountTests.py" for 10 times, and "losetup -a" should give empty result after each run. When gvfs is running, "losetup -a" would show 2 new loop devices are occupied after each run.
gvfs is not installed on the machines, so I doubt this is the same error :( I suggest to add some cleanup code at the end of the tests to free all the used loop devices (if you free only the ones you use, it will not fail when running parallel tests).
Adding more loop devices or cleaning them all manually is only a temporary solution.
Thanks!
From what I can there is 3 places would consume a loop device. In
tests/parted_utils_tests.py it calls losetup directly and tears down the loop device at the end of each test. In tests/mountTests.py and tests/mkimageTests.py, it consumes loop device via loop mount, and umount the device so the loop device is freed automatically. All three places free loop device at the end of the test, unless there is exception or test fails. In this case we can catch that exception and free the device.
Another reason could be other process occupying the mount point or loop device, in this case trying to free it again will fail and can not solve this problem. So I think we should investigate why it is not released and locate the bug. Could you lsof the mount point and the device to see if any process is occupying them? And could you try to run parted_utils_tests.py, mountTests.py and mkimageTests.py separately and see "losetup -a"'s report, to see which of the test module consumes but not free the device?
-- David Caro
Red Hat Czech s.r.o. Continuous Integration Engineer - EMEA ENG Virtualization R&D
Tel.: +420 532 294 605 Email: dcaro@redhat.com Web: www.cz.redhat.com Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno, Czech Republic RHT Global #: 82-62605
vdsm-devel@lists.fedorahosted.org