* Dan Kenigsberg danken@redhat.com [2012-08-16 03:37]:
Dan Kenigsberg has posted comments on this change.
Change subject: Fix vdsm-faqemu VM launch failure on hosts with newer libvirt ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File vdsm_hooks/faqemu/vdsm-faqemu Line 20: Line 21: qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) Line 22: # libvirt 0.9.10+ checks for the presence of /dev/kvm to provide Line 23: # better error messages, but this breaks this hook, fake it out Line 24: os.system("touch /dev/kvm") how can this work? this script runs as vdsm.
# sudo -u vdsm touch /dev/kvm touch: cannot touch `/dev/kvm': Permission denied
But I can assure you that it did work. I didn't just touch the file as root. I installed the faqemu hook and /dev/kvm is owned by root. Are you sure hooks run as vdsm?
It's run as root:
[root@ichigo-dom228 tmp]# cat /usr/libexec/vdsm/vdsm-faqemu #!/usr/bin/python
import os import sys
qemu_paths = [ "/usr/bin/qemu-kvm", "/usr/libexec/qemu-kvm", ]
for qemu_bin in qemu_paths: if os.path.exists(qemu_bin): break else: raise RuntimeError, "QEMU binary not found"
qemu_argv = [ qemu_bin ]
if not "-no-kvm" in sys.argv: qemu_argv += [ "-no-kvm" ]
qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) os.system('id > /tmp/faqemu_id.out') os.system('touch /dev/kvm') os.execv(qemu_bin, qemu_argv)
[root@ichigo-dom228 tmp]# cat /tmp/faqemu_id.out uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:virtd_t:s0-s0:c0.c1023
also, there are less expensive ways to create a file - how about shipping it in this hook's rpm ?
Is that really less expensive? I worry about having an rpm include the file; and /dev gets remounted/changed accross reboots; so this wouldn't work.
What's expensive about a touch?
-- To view, visit http://gerrit.ovirt.org/7223 To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: comment Gerrit-Change-Id: I1dc88f2931e8f716c1732a51d9f65f600f4fd1ef Gerrit-PatchSet: 1 Gerrit-Project: vdsm Gerrit-Branch: master Gerrit-Owner: Ryan Harper ryanh@us.ibm.com Gerrit-Reviewer: Dan Kenigsberg danken@redhat.com Gerrit-Reviewer: Mark Wu wudxw@linux.vnet.ibm.com Gerrit-Reviewer: Ryan Harper ryanh@us.ibm.com Gerrit-Reviewer: oVirt Jenkins CI Server
On Thu, Aug 16, 2012 at 10:53:17AM -0500, Ryan Harper wrote:
- Dan Kenigsberg danken@redhat.com [2012-08-16 03:37]:
Dan Kenigsberg has posted comments on this change.
Change subject: Fix vdsm-faqemu VM launch failure on hosts with newer libvirt ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File vdsm_hooks/faqemu/vdsm-faqemu Line 20: Line 21: qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) Line 22: # libvirt 0.9.10+ checks for the presence of /dev/kvm to provide Line 23: # better error messages, but this breaks this hook, fake it out Line 24: os.system("touch /dev/kvm") how can this work? this script runs as vdsm.
# sudo -u vdsm touch /dev/kvm touch: cannot touch `/dev/kvm': Permission denied
But I can assure you that it did work. I didn't just touch the file as root. I installed the faqemu hook and /dev/kvm is owned by root. Are you sure hooks run as vdsm?
Yes. They are called by hooks.py's execCmd. However I was wrong about this script being a hook script: it is the emulator used by libvirt instead of qemu. Again, I am very much surprised that this runs as root - libvirt's default is qemu:qemu.
It's run as root:
[root@ichigo-dom228 tmp]# cat /usr/libexec/vdsm/vdsm-faqemu #!/usr/bin/python
import os import sys
qemu_paths = [ "/usr/bin/qemu-kvm", "/usr/libexec/qemu-kvm", ]
for qemu_bin in qemu_paths: if os.path.exists(qemu_bin): break else: raise RuntimeError, "QEMU binary not found"
qemu_argv = [ qemu_bin ]
if not "-no-kvm" in sys.argv: qemu_argv += [ "-no-kvm" ]
qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) os.system('id > /tmp/faqemu_id.out') os.system('touch /dev/kvm') os.execv(qemu_bin, qemu_argv)
[root@ichigo-dom228 tmp]# cat /tmp/faqemu_id.out uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:virtd_t:s0-s0:c0.c1023
also, there are less expensive ways to create a file - how about shipping it in this hook's rpm ?
Is that really less expensive? I worry about having an rpm include the file; and /dev gets remounted/changed accross reboots; so this wouldn't work.
What's expensive about a touch?
It is, comparing to opening the file for writing
file('/dev/kvm', 'w')
Anyway, I cannot say that I'm happy with this solution: nothing clears /dev/kvm after the hook is uninstalled, for one's.
Regards,
Dan.
* Dan Kenigsberg danken@redhat.com [2012-08-16 15:59]:
On Thu, Aug 16, 2012 at 10:53:17AM -0500, Ryan Harper wrote:
- Dan Kenigsberg danken@redhat.com [2012-08-16 03:37]:
Dan Kenigsberg has posted comments on this change.
Change subject: Fix vdsm-faqemu VM launch failure on hosts with newer libvirt ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File vdsm_hooks/faqemu/vdsm-faqemu Line 20: Line 21: qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) Line 22: # libvirt 0.9.10+ checks for the presence of /dev/kvm to provide Line 23: # better error messages, but this breaks this hook, fake it out Line 24: os.system("touch /dev/kvm") how can this work? this script runs as vdsm.
# sudo -u vdsm touch /dev/kvm touch: cannot touch `/dev/kvm': Permission denied
But I can assure you that it did work. I didn't just touch the file as root. I installed the faqemu hook and /dev/kvm is owned by root. Are you sure hooks run as vdsm?
Yes. They are called by hooks.py's execCmd. However I was wrong about this script being a hook script: it is the emulator used by libvirt instead of qemu. Again, I am very much surprised that this runs as root - libvirt's default is qemu:qemu.
Agreed, I'm not quite sure what's going on here as the VMs indeed are running as non-root qemu user:
qemu 31698 2.0 0.8 997456 71884 ? Sl Aug16 25:45 /usr/bin/qemu-kvm -no-kvm -S -M pc-0.14 -m 20 -smp 1,sockets=1,cores=1,threads=1 -name vm2 -uuid 0c3767f1-e525-433c-aeaf-e27921624a37 -smbios type=1,manufacturer=Red Hat,product=RHEV Hypervisor,version=17-1,serial=69C248FD-BFE5-35FE-EB8E-1531D910109A_00:ff:fe:00:00:e4,uuid=0c3767f1-e525-433c-aeaf-e27921624a37 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm2.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2012-08-16T15:43:58,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/2a8898a0-e6ea-11e1-985c-00fffe0000eb/7af50e73-bdc4-4814-84bc-dd802542d688/images/11111111-1111-1111-1111-111111111111/RHEL6.3-20120531.0-Server-x86_64-DVD1.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/rhev/data-center/2a8898a0-e6ea-11e1-985c-00fffe0000eb/f01871d2-de00-43a7-93e4-275470245cc9/images/b0139761-8926-42b6-8de6-aebb400190a6/3b3870af-4bbb-4c24-b438-f8e6c59f53fe,if=none,id=drive-virtio-disk0,format=raw,serial=b0139761-8926-42b6-8de6-aebb400190a6,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:a8:44:01,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/vm2.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -chardev pty,id=charconsole0 -device virtconsole,chardev=charconsole0,id=console0 -spice port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record -k en-us -vga qxl -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
It's run as root:
[root@ichigo-dom228 tmp]# cat /usr/libexec/vdsm/vdsm-faqemu #!/usr/bin/python
import os import sys
qemu_paths = [ "/usr/bin/qemu-kvm", "/usr/libexec/qemu-kvm", ]
for qemu_bin in qemu_paths: if os.path.exists(qemu_bin): break else: raise RuntimeError, "QEMU binary not found"
qemu_argv = [ qemu_bin ]
if not "-no-kvm" in sys.argv: qemu_argv += [ "-no-kvm" ]
qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) os.system('id > /tmp/faqemu_id.out') os.system('touch /dev/kvm') os.execv(qemu_bin, qemu_argv)
[root@ichigo-dom228 tmp]# cat /tmp/faqemu_id.out uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:virtd_t:s0-s0:c0.c1023
also, there are less expensive ways to create a file - how about shipping it in this hook's rpm ?
Is that really less expensive? I worry about having an rpm include the file; and /dev gets remounted/changed accross reboots; so this wouldn't work.
What's expensive about a touch?
It is, comparing to opening the file for writing
file('/dev/kvm', 'w')
We're launching a VM, an additional exec isn't a lot of overhead. If someone comes back and says launching the VM took too long and we spend all our time in one os.system('touch') call, then maybe you're right.
(platechiller) ~ % cat touch.py import os os.system('touch /dev/kvm') (platechiller) ~ % sudo time python touch.py 0.01user 0.00system 0:00.01elapsed 84%CPU (0avgtext+0avgdata 21328maxresident)k 0inputs+0outputs (0major+2424minor)pagefaults 0swaps
Anyway, I cannot say that I'm happy with this solution: nothing clears /dev/kvm after the hook is uninstalled, for one's.
/dev/kvm will be removed on reboot since /dev is reconstructed.
The presence of the file doesn't bother loading of the module either:
(platechiller) ~ % ls -al /dev/kvm crw-rw----+ 1 root kvm 10, 232 Aug 17 08:06 /dev/kvm (platechiller) ~ % sudo modprobe -vr kvm_intel kvm rmmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm-intel.ko rmmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm.ko (platechiller) ~ % ls -al /dev/kvm ls: cannot access /dev/kvm: No such file or directory (platechiller) ~ % sudo touch /dev/kvm (platechiller) ~ % ls -al /dev/kvm -rw-r--r-- 1 root root 0 Aug 17 08:06 /dev/kvm (platechiller) ~ % sudo modprobe -v kvm-intel insmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm.ko insmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm-intel.ko (platechiller) ~ % ls -al /dev/kvm crw-rw----+ 1 root kvm 10, 232 Aug 17 08:07 /dev/kvm
Regards,
Dan.
On Fri, Aug 17, 2012 at 08:10:40AM -0500, Ryan Harper wrote:
- Dan Kenigsberg danken@redhat.com [2012-08-16 15:59]:
On Thu, Aug 16, 2012 at 10:53:17AM -0500, Ryan Harper wrote:
- Dan Kenigsberg danken@redhat.com [2012-08-16 03:37]:
Dan Kenigsberg has posted comments on this change.
Change subject: Fix vdsm-faqemu VM launch failure on hosts with newer libvirt ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File vdsm_hooks/faqemu/vdsm-faqemu Line 20: Line 21: qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) Line 22: # libvirt 0.9.10+ checks for the presence of /dev/kvm to provide Line 23: # better error messages, but this breaks this hook, fake it out Line 24: os.system("touch /dev/kvm") how can this work? this script runs as vdsm.
# sudo -u vdsm touch /dev/kvm touch: cannot touch `/dev/kvm': Permission denied
But I can assure you that it did work. I didn't just touch the file as root. I installed the faqemu hook and /dev/kvm is owned by root. Are you sure hooks run as vdsm?
Yes. They are called by hooks.py's execCmd. However I was wrong about this script being a hook script: it is the emulator used by libvirt instead of qemu. Again, I am very much surprised that this runs as root - libvirt's default is qemu:qemu.
Agreed, I'm not quite sure what's going on here as the VMs indeed are running as non-root qemu user:
Could libvirt be running the emulator twice? Once as root to check -help, and then as qemu?
qemu 31698 2.0 0.8 997456 71884 ? Sl Aug16 25:45 /usr/bin/qemu-kvm
It's run as root:
[root@ichigo-dom228 tmp]# cat /usr/libexec/vdsm/vdsm-faqemu #!/usr/bin/python
import os import sys
qemu_paths = [ "/usr/bin/qemu-kvm", "/usr/libexec/qemu-kvm", ]
for qemu_bin in qemu_paths: if os.path.exists(qemu_bin): break else: raise RuntimeError, "QEMU binary not found"
qemu_argv = [ qemu_bin ]
if not "-no-kvm" in sys.argv: qemu_argv += [ "-no-kvm" ]
qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) os.system('id > /tmp/faqemu_id.out') os.system('touch /dev/kvm') os.execv(qemu_bin, qemu_argv)
[root@ichigo-dom228 tmp]# cat /tmp/faqemu_id.out uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:virtd_t:s0-s0:c0.c1023
also, there are less expensive ways to create a file - how about shipping it in this hook's rpm ?
Is that really less expensive? I worry about having an rpm include the file; and /dev gets remounted/changed accross reboots; so this wouldn't work.
What's expensive about a touch?
It is, comparing to opening the file for writing
file('/dev/kvm', 'w')
We're launching a VM, an additional exec isn't a lot of overhead. If someone comes back and says launching the VM took too long and we spend all our time in one os.system('touch') call, then maybe you're right.
(platechiller) ~ % cat touch.py import os os.system('touch /dev/kvm') (platechiller) ~ % sudo time python touch.py 0.01user 0.00system 0:00.01elapsed 84%CPU (0avgtext+0avgdata 21328maxresident)k 0inputs+0outputs (0major+2424minor)pagefaults 0swaps
Yes, one fork is negligible. But they pile up. I see no reason to call fork(2) when you actually want open(2).
Anyway, I cannot say that I'm happy with this solution: nothing clears /dev/kvm after the hook is uninstalled, for one's.
/dev/kvm will be removed on reboot since /dev is reconstructed.
The presence of the file doesn't bother loading of the module either:
(platechiller) ~ % ls -al /dev/kvm crw-rw----+ 1 root kvm 10, 232 Aug 17 08:06 /dev/kvm (platechiller) ~ % sudo modprobe -vr kvm_intel kvm rmmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm-intel.ko rmmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm.ko (platechiller) ~ % ls -al /dev/kvm ls: cannot access /dev/kvm: No such file or directory (platechiller) ~ % sudo touch /dev/kvm (platechiller) ~ % ls -al /dev/kvm -rw-r--r-- 1 root root 0 Aug 17 08:06 /dev/kvm (platechiller) ~ % sudo modprobe -v kvm-intel insmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm.ko insmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm-intel.ko (platechiller) ~ % ls -al /dev/kvm crw-rw----+ 1 root kvm 10, 232 Aug 17 08:07 /dev/kvm
It's less destructive than I thought, but it's not prettier... On the other hand, it let's touch the file only when the script is enabled.
Dan.
* Dan Kenigsberg danken@redhat.com [2012-08-19 07:36]:
On Fri, Aug 17, 2012 at 08:10:40AM -0500, Ryan Harper wrote:
- Dan Kenigsberg danken@redhat.com [2012-08-16 15:59]:
On Thu, Aug 16, 2012 at 10:53:17AM -0500, Ryan Harper wrote:
- Dan Kenigsberg danken@redhat.com [2012-08-16 03:37]:
Dan Kenigsberg has posted comments on this change.
Change subject: Fix vdsm-faqemu VM launch failure on hosts with newer libvirt ......................................................................
Patch Set 1: I would prefer that you didn't submit this
(1 inline comment)
.................................................... File vdsm_hooks/faqemu/vdsm-faqemu Line 20: Line 21: qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) Line 22: # libvirt 0.9.10+ checks for the presence of /dev/kvm to provide Line 23: # better error messages, but this breaks this hook, fake it out Line 24: os.system("touch /dev/kvm") how can this work? this script runs as vdsm.
# sudo -u vdsm touch /dev/kvm touch: cannot touch `/dev/kvm': Permission denied
But I can assure you that it did work. I didn't just touch the file as root. I installed the faqemu hook and /dev/kvm is owned by root. Are you sure hooks run as vdsm?
Yes. They are called by hooks.py's execCmd. However I was wrong about this script being a hook script: it is the emulator used by libvirt instead of qemu. Again, I am very much surprised that this runs as root - libvirt's default is qemu:qemu.
Agreed, I'm not quite sure what's going on here as the VMs indeed are running as non-root qemu user:
Could libvirt be running the emulator twice? Once as root to check -help, and then as qemu?
I was just thinking that, but I'm pretty sure we'd throw an error on access.
(platechiller) ~ % sudo bash [sudo] password for rharper: (platechiller) tmp % touch /dev/kvmtest (platechiller) tmp % ls -al /dev/kvmtest -rw-r--r-- 1 root root 0 Aug 20 08:37 /dev/kvmtest (platechiller) tmp % exit exit (platechiller) ~ % whoami rharper (platechiller) ~ % touch /dev/kvmtest touch: cannot touch `/dev/kvmtest': Permission denied
qemu 31698 2.0 0.8 997456 71884 ? Sl Aug16 25:45 /usr/bin/qemu-kvm
It's run as root:
[root@ichigo-dom228 tmp]# cat /usr/libexec/vdsm/vdsm-faqemu #!/usr/bin/python
import os import sys
qemu_paths = [ "/usr/bin/qemu-kvm", "/usr/libexec/qemu-kvm", ]
for qemu_bin in qemu_paths: if os.path.exists(qemu_bin): break else: raise RuntimeError, "QEMU binary not found"
qemu_argv = [ qemu_bin ]
if not "-no-kvm" in sys.argv: qemu_argv += [ "-no-kvm" ]
qemu_argv += filter(lambda x: x != "-enable-kvm", sys.argv[1:]) os.system('id > /tmp/faqemu_id.out') os.system('touch /dev/kvm') os.execv(qemu_bin, qemu_argv)
[root@ichigo-dom228 tmp]# cat /tmp/faqemu_id.out uid=0(root) gid=0(root) groups=0(root) context=system_u:system_r:virtd_t:s0-s0:c0.c1023
also, there are less expensive ways to create a file - how about shipping it in this hook's rpm ?
Is that really less expensive? I worry about having an rpm include the file; and /dev gets remounted/changed accross reboots; so this wouldn't work.
What's expensive about a touch?
It is, comparing to opening the file for writing
file('/dev/kvm', 'w')
We're launching a VM, an additional exec isn't a lot of overhead. If someone comes back and says launching the VM took too long and we spend all our time in one os.system('touch') call, then maybe you're right.
(platechiller) ~ % cat touch.py import os os.system('touch /dev/kvm') (platechiller) ~ % sudo time python touch.py 0.01user 0.00system 0:00.01elapsed 84%CPU (0avgtext+0avgdata 21328maxresident)k 0inputs+0outputs (0major+2424minor)pagefaults 0swaps
Yes, one fork is negligible. But they pile up. I see no reason to call fork(2) when you actually want open(2).
I still fail to see the significance of the overhead here. we're already in an exceptional path
1) I had to install faqemu hook 2) I had to be running on a host without kvm (going to be slow anyhow) 3) and this is during VM creation, in which libvirt already execs qemu-kvm no less than 3 times.
Since I'd really like to get VM creation working for me, I'll respin.
Anyway, I cannot say that I'm happy with this solution: nothing clears /dev/kvm after the hook is uninstalled, for one's.
/dev/kvm will be removed on reboot since /dev is reconstructed.
The presence of the file doesn't bother loading of the module either:
(platechiller) ~ % ls -al /dev/kvm crw-rw----+ 1 root kvm 10, 232 Aug 17 08:06 /dev/kvm (platechiller) ~ % sudo modprobe -vr kvm_intel kvm rmmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm-intel.ko rmmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm.ko (platechiller) ~ % ls -al /dev/kvm ls: cannot access /dev/kvm: No such file or directory (platechiller) ~ % sudo touch /dev/kvm (platechiller) ~ % ls -al /dev/kvm -rw-r--r-- 1 root root 0 Aug 17 08:06 /dev/kvm (platechiller) ~ % sudo modprobe -v kvm-intel insmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm.ko insmod /lib/modules/3.2.0-29-generic/kernel/arch/x86/kvm/kvm-intel.ko (platechiller) ~ % ls -al /dev/kvm crw-rw----+ 1 root kvm 10, 232 Aug 17 08:07 /dev/kvm
It's less destructive than I thought, but it's not prettier... On the other hand, it let's touch the file only when the script is enabled.
Dan.
vdsm-devel@lists.fedorahosted.org