Hi all,
I am trying to create a VM in vdsm, but I find that selinux is preventing qemu from visiting images in vdsm storage. I add two policies to selinux then start the VM successfully. Not sure if I've done it right, so I'd like to ask for some advice.
The directory structure of my vdsm storage is as follow:
$ tree -l /rhev /rhev `-- data-center |-- 3ace0f74-a9fa-11e1-bb33-00247edb4743 | |-- 1de3f4e2-a9f9-11e1-9956-00247edb4743 -> /rhev/data-center/mnt/_teststorage/1de3f4e2-a9f9-11e1-9956-00247edb4743 | | |-- dom_md | | | |-- ids | | | |-- inbox | | | |-- leases | | | |-- metadata | | | `-- outbox | | |-- images | | | `-- 8230cd4a-aa1b-11e1-970c-00247edb4743 | | | |-- b9e163b6-aa1c-11e1-94f6-00247edb4743 | | | `-- b9e163b6-aa1c-11e1-94f6-00247edb4743.meta | | `-- master | | |-- tasks | | | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.job.0 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.0 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.1 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.result | | | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.task | | `-- vms | `-- mastersd -> /rhev/data-center/mnt/_teststorage/1de3f4e2-a9f9-11e1-9956-00247edb4743 [recursive, not followed] |-- hsm-tasks `-- mnt `-- _teststorage -> /teststorage `-- 1de3f4e2-a9f9-11e1-9956-00247edb4743 |-- dom_md | |-- ids | |-- inbox | |-- leases | |-- metadata | `-- outbox |-- images | `-- 8230cd4a-aa1b-11e1-970c-00247edb4743 | |-- b9e163b6-aa1c-11e1-94f6-00247edb4743 | `-- b9e163b6-aa1c-11e1-94f6-00247edb4743.meta `-- master |-- tasks | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.job.0 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.0 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.1 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.result | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.task `-- vms
22 directories, 24 files
The storage is constructed using vdsClient. I create a LOCALFS DATA domain, a pool, an image and a volume.
I find that libvirtd can relable the volume to "system_u:object_r:svirt_image_t:s0:cXXX,cXXX", and it starts qemu as "system_u:system_r:svirt_t:s0:cXXX,cXXX"
$ ll -Z b9e163b6-aa1c-11e1-94f6-00247edb4743 -rw-rw-r--. vdsm kvm system_u:object_r:svirt_image_t:s0:c248,c603 b9e163b6-aa1c-11e1-94f6-00247edb4743 $ ps auxZ | grep qemu system_u:system_r:svirt_t:s0:c248,c603 qemu 11132 60.0 6.9 1531336 273756 ? Sl 13:58 0:31 /usr/bin/qemu-kvm ...(lots of arguments)
This is an expected feature of sVirt, and qemu is supposed to have access to the volume. To access the volume file "b9e163b6-aa1c-11e1-94f6-00247edb4743", qemu must go through "/rhev/data-center/3ace0f74-a9fa-11e1-bb33-00247edb4743/1de3f4e2-a9f9-11e1-9956-00247edb4743/images/8230cd4a-aa1b-11e1-970c-00247edb4743/", so it must read the soft link "1de3f4e2-..." and "_teststorage". However, these two soft links are created by vdsm with security label "default_t", and qemu is labeled as "svirt_t", so selinux is preventing qemu reading the soft links, this means qemu can not access the volume file.
I lookup the libvirt documentation, it says libvirt runs qemu in a confined domain, only allowing qemu to access files and devices labeled as "system_u:object_r:virt_image_t" or "system_u:object_r:svirt_image_t" (http://libvirt.org/drvqemu.html#securityselinux). So I add two policies as follow:
semanage fcontext -a -t virt_image_t '/rhev/data-center/mnt/[^/]+' semanage fcontext -a -t virt_image_t '/rhev/data-center/[-0-9a-f]+/[-0-9a-f]+'
This let qemu to access those soft links.
Including the existing policies policies on /rhev, now the overall policies are as follow:
# semanage fcontext -l | grep '^/rhev' /rhev directory system_u:object_r:mnt_t:s0 /rhev(/[^/]*)? directory system_u:object_r:mnt_t:s0 /rhev/[^/]*/.* all files <<None>> /rhev/data-center/[-0-9a-f]+/[-0-9a-f]+ all files system_u:object_r:virt_image_t:s0 /rhev/data-center/mnt/[^/]+ all files system_u:object_r:virt_image_t:s0
Then I "restorecon -Rv /rhev". After doing this, I can create and boot VM in vdsm now.
Now I have three questions. 1. I think I'm giving qemu the least privilege it needs, but I'm not sure if the policies are too permissive. 2. Does anyone ever have the same problem? Am I solving it in a right way? 3. Since the soft links are created by vdsm, this seems like a bug. If I want to submit a patch for this bug, which project should I submit those new policies: selinux, libvirt or vdsm?
On 06/11/2012 02:56 PM, Zhou Zheng Sheng wrote:
Hi all,
I am trying to create a VM in vdsm, but I find that selinux is preventing qemu from visiting images in vdsm storage. I add two policies to selinux then start the VM successfully. Not sure if I've done it right, so I'd like to ask for some advice.
The directory structure of my vdsm storage is as follow:
$ tree -l /rhev /rhev `-- data-center |-- 3ace0f74-a9fa-11e1-bb33-00247edb4743 | |-- 1de3f4e2-a9f9-11e1-9956-00247edb4743 -> /rhev/data-center/mnt/_teststorage/1de3f4e2-a9f9-11e1-9956-00247edb4743 | | |-- dom_md | | | |-- ids | | | |-- inbox | | | |-- leases | | | |-- metadata | | | `-- outbox | | |-- images | | | `-- 8230cd4a-aa1b-11e1-970c-00247edb4743 | | | |-- b9e163b6-aa1c-11e1-94f6-00247edb4743 | | | `-- b9e163b6-aa1c-11e1-94f6-00247edb4743.meta | | `-- master | | |-- tasks | | | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.job.0 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.0 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.1 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.result | | | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.task | | `-- vms | `-- mastersd -> /rhev/data-center/mnt/_teststorage/1de3f4e2-a9f9-11e1-9956-00247edb4743 [recursive, not followed] |-- hsm-tasks `-- mnt `-- _teststorage -> /teststorage `-- 1de3f4e2-a9f9-11e1-9956-00247edb4743 |-- dom_md | |-- ids | |-- inbox | |-- leases | |-- metadata | `-- outbox |-- images | `-- 8230cd4a-aa1b-11e1-970c-00247edb4743 | |-- b9e163b6-aa1c-11e1-94f6-00247edb4743 | `-- b9e163b6-aa1c-11e1-94f6-00247edb4743.meta `-- master |-- tasks | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.job.0 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.0 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.1 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.result | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.task `-- vms
22 directories, 24 files
The storage is constructed using vdsClient. I create a LOCALFS DATA domain, a pool, an image and a volume.
I find that libvirtd can relable the volume to "system_u:object_r:svirt_image_t:s0:cXXX,cXXX", and it starts qemu as "system_u:system_r:svirt_t:s0:cXXX,cXXX"
$ ll -Z b9e163b6-aa1c-11e1-94f6-00247edb4743 -rw-rw-r--. vdsm kvm system_u:object_r:svirt_image_t:s0:c248,c603 b9e163b6-aa1c-11e1-94f6-00247edb4743 $ ps auxZ | grep qemu system_u:system_r:svirt_t:s0:c248,c603 qemu 11132 60.0 6.9 1531336 273756 ? Sl 13:58 0:31 /usr/bin/qemu-kvm ...(lots of arguments)
This is an expected feature of sVirt, and qemu is supposed to have access to the volume. To access the volume file "b9e163b6-aa1c-11e1-94f6-00247edb4743", qemu must go through "/rhev/data-center/3ace0f74-a9fa-11e1-bb33-00247edb4743/1de3f4e2-a9f9-11e1-9956-00247edb4743/images/8230cd4a-aa1b-11e1-970c-00247edb4743/", so it must read the soft link "1de3f4e2-..." and "_teststorage". However, these two soft links are created by vdsm with security label "default_t", and qemu is labeled as "svirt_t", so selinux is preventing qemu reading the soft links, this means qemu can not access the volume file.
I lookup the libvirt documentation, it says libvirt runs qemu in a confined domain, only allowing qemu to access files and devices labeled as "system_u:object_r:virt_image_t" or "system_u:object_r:svirt_image_t" (http://libvirt.org/drvqemu.html#securityselinux). So I add two policies as follow:
semanage fcontext -a -t virt_image_t '/rhev/data-center/mnt/[^/]+' semanage fcontext -a -t virt_image_t '/rhev/data-center/[-0-9a-f]+/[-0-9a-f]+'
This let qemu to access those soft links.
Including the existing policies policies on /rhev, now the overall policies are as follow:
# semanage fcontext -l | grep '^/rhev' /rhev directory system_u:object_r:mnt_t:s0 /rhev(/[^/]*)? directory system_u:object_r:mnt_t:s0 /rhev/[^/]*/.* all files<<None>> /rhev/data-center/[-0-9a-f]+/[-0-9a-f]+ all files system_u:object_r:virt_image_t:s0 /rhev/data-center/mnt/[^/]+ all files system_u:object_r:virt_image_t:s0
Then I "restorecon -Rv /rhev". After doing this, I can create and boot VM in vdsm now.
Now I have three questions.
- I think I'm giving qemu the least privilege it needs, but I'm not
sure if the policies are too permissive. 2. Does anyone ever have the same problem? Am I solving it in a right way? 3. Since the soft links are created by vdsm, this seems like a bug. If I want to submit a patch for this bug, which project should I submit those new policies: selinux, libvirt or vdsm?
Hi Zhengsheng,
In my test, the SELinux type of link files created by vdsm is "mnt_t", which inherits from their parent folders. That's the default SELinux behaviour. So could you check how the type become 'default_t' or let me know how to reproduce the problem?
Mark.
Thanks Mark. I restore all the original policies, delete /rhev, and reinstall vdsm. I find out the soft links are created with "mnt_t" label and everything is OK now.
In my previous setup, there was some problems installing and starting vdsm, it complained me that an error accessing /rhev/data-center/mnt. Here is the log:
MainThread::INFO::2012-04-19 12:00:31,932::vdsm::70::vds::(run) I am the actual vdsm 4.9-0 MainThread::DEBUG::2012-04-19 12:00:32,877::resourceManager::379::ResourceManager::(registerNamespace) Registering namespace 'Storage' MainThread::DEBUG::2012-04-19 12:00:32,878::threadPool::45::Misc.ThreadPool::(__init__) Enter - numThreads: 10.0, waitTimeout: 3, maxTasks: 500.0 MainThread::ERROR::2012-04-19 12:00:32,889::clientIF::162::vds::(_initIRS) Error initializing IRS Traceback (most recent call last): File "/usr/share/vdsm/clientIF.py", line 160, in _initIRS self.irs = Dispatcher(HSM()) File "/usr/share/vdsm/storage/hsm.py", line 282, in __init__ fileUtils.createdir(mountBasePath) File "/usr/share/vdsm/storage/fileUtils.py", line 149, in createdir os.makedirs(dirPath) File "/usr/lib64/python2.7/os.py", line 157, in makedirs mkdir(name, mode) OSError: [Errno 13] Permission denied: '/rhev/data-center/mnt'
So I created the folders manually. The first installation was two month ago, I can not remember the operating details now.
When I created those folders, I had not learned how to mess with selinux yet, so they were supposed to have the correct "mnt_t" label. Then I did some storage experiments using vdsClient. Last week I began to do some VM experiments and found some permission errors. That was when I learned selinux commands and began to investigate the problem. Today I lookup various log files, still can not find the clues about why the label was wrong.
Anyway, deleting everything and reinstalling vdsm solve the problem. I will keep an eye on those labels. Thanks again!
On 06/12/2012 13:47, Mark Wu wrote:
On 06/11/2012 02:56 PM, Zhou Zheng Sheng wrote:
Hi all,
I am trying to create a VM in vdsm, but I find that selinux is preventing qemu from visiting images in vdsm storage. I add two policies to selinux then start the VM successfully. Not sure if I've done it right, so I'd like to ask for some advice.
The directory structure of my vdsm storage is as follow:
$ tree -l /rhev /rhev `-- data-center |-- 3ace0f74-a9fa-11e1-bb33-00247edb4743 | |-- 1de3f4e2-a9f9-11e1-9956-00247edb4743 -> /rhev/data-center/mnt/_teststorage/1de3f4e2-a9f9-11e1-9956-00247edb4743 | | |-- dom_md | | | |-- ids | | | |-- inbox | | | |-- leases | | | |-- metadata | | | `-- outbox | | |-- images | | | `-- 8230cd4a-aa1b-11e1-970c-00247edb4743 | | | |-- b9e163b6-aa1c-11e1-94f6-00247edb4743 | | | `-- b9e163b6-aa1c-11e1-94f6-00247edb4743.meta | | `-- master | | |-- tasks | | | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.job.0 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.0 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.1 | | | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.result | | | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.task | | `-- vms | `-- mastersd -> /rhev/data-center/mnt/_teststorage/1de3f4e2-a9f9-11e1-9956-00247edb4743 [recursive, not followed] |-- hsm-tasks `-- mnt `-- _teststorage -> /teststorage `-- 1de3f4e2-a9f9-11e1-9956-00247edb4743 |-- dom_md | |-- ids | |-- inbox | |-- leases | |-- metadata | `-- outbox |-- images | `-- 8230cd4a-aa1b-11e1-970c-00247edb4743 | |-- b9e163b6-aa1c-11e1-94f6-00247edb4743 | `-- b9e163b6-aa1c-11e1-94f6-00247edb4743.meta `-- master |-- tasks | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.job.0 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.0 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.recover.1 | |-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.result | `-- a01abf82-c8f2-4d52-8e50-f26a2a2994b5.task `-- vms
22 directories, 24 files
The storage is constructed using vdsClient. I create a LOCALFS DATA domain, a pool, an image and a volume.
I find that libvirtd can relable the volume to "system_u:object_r:svirt_image_t:s0:cXXX,cXXX", and it starts qemu as "system_u:system_r:svirt_t:s0:cXXX,cXXX"
$ ll -Z b9e163b6-aa1c-11e1-94f6-00247edb4743 -rw-rw-r--. vdsm kvm system_u:object_r:svirt_image_t:s0:c248,c603 b9e163b6-aa1c-11e1-94f6-00247edb4743 $ ps auxZ | grep qemu system_u:system_r:svirt_t:s0:c248,c603 qemu 11132 60.0 6.9 1531336 273756 ? Sl 13:58 0:31 /usr/bin/qemu-kvm ...(lots of arguments)
This is an expected feature of sVirt, and qemu is supposed to have access to the volume. To access the volume file "b9e163b6-aa1c-11e1-94f6-00247edb4743", qemu must go through "/rhev/data-center/3ace0f74-a9fa-11e1-bb33-00247edb4743/1de3f4e2-a9f9-11e1-9956-00247edb4743/images/8230cd4a-aa1b-11e1-970c-00247edb4743/",
so it must read the soft link "1de3f4e2-..." and "_teststorage". However, these two soft links are created by vdsm with security label "default_t", and qemu is labeled as "svirt_t", so selinux is preventing qemu reading the soft links, this means qemu can not access the volume file.
I lookup the libvirt documentation, it says libvirt runs qemu in a confined domain, only allowing qemu to access files and devices labeled as "system_u:object_r:virt_image_t" or "system_u:object_r:svirt_image_t" (http://libvirt.org/drvqemu.html#securityselinux). So I add two policies as follow:
semanage fcontext -a -t virt_image_t '/rhev/data-center/mnt/[^/]+' semanage fcontext -a -t virt_image_t '/rhev/data-center/[-0-9a-f]+/[-0-9a-f]+'
This let qemu to access those soft links.
Including the existing policies policies on /rhev, now the overall policies are as follow:
# semanage fcontext -l | grep '^/rhev' /rhev directory system_u:object_r:mnt_t:s0 /rhev(/[^/]*)? directory system_u:object_r:mnt_t:s0 /rhev/[^/]*/.* all files<<None>> /rhev/data-center/[-0-9a-f]+/[-0-9a-f]+ all files system_u:object_r:virt_image_t:s0 /rhev/data-center/mnt/[^/]+ all files system_u:object_r:virt_image_t:s0
Then I "restorecon -Rv /rhev". After doing this, I can create and boot VM in vdsm now.
Now I have three questions.
- I think I'm giving qemu the least privilege it needs, but I'm not
sure if the policies are too permissive. 2. Does anyone ever have the same problem? Am I solving it in a right way? 3. Since the soft links are created by vdsm, this seems like a bug. If I want to submit a patch for this bug, which project should I submit those new policies: selinux, libvirt or vdsm?
Hi Zhengsheng,
In my test, the SELinux type of link files created by vdsm is "mnt_t", which inherits from their parent folders. That's the default SELinux behaviour. So could you check how the type become 'default_t' or let me know how to reproduce the problem?
Mark.
vdsm-devel@lists.fedorahosted.org