Hi,
In the VDSM code about some SPM operations like HSM.deleteImage(), It is found that VDSM doesn't check if the operation will be launched on a SPM host or not. It only checks if the storage pool is already acquired by one SPM host, but not necessary the same host as the SPM operation is delivered to. The code is like this:
HSM.deleteImage() { ... HSM._spmSchedule() { self.validateSPM(spUUID) <--- Only check if the storage pool was acquired by one host, but not necessary this host } ... }
So it really depends on the node management application AKA ovirt-engine to dispatch the SPM operations to the right VDSM host. And the VDSM host itself doesn't check if it is the SPM host which can execute the operation. To me, it is a bit broken. When the engine query the VDSM host who is the SPM host, it can get the right one. However, the host may be broken for some reason after the engine believes it is the SPM host and the host loses the SPM privilege, another host will take the SPM role. Then the engine continue to send the SPM operations to the broken host. As a result, the SPM operation will be launched on a non-SPM host. So I think there is a small window of racing to corrupt the VDSM hosts meta data. I think VDSM host should check if it is SPM before the SPM job is scheduled. If the host lost the SPM role already, It should fail the RPC call from the engine to let the engine to retry the operation after engine knows the failure of the former call.
----- Original Message -----
From: "Shu Ming" shuming@linux.vnet.ibm.com To: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Tuesday, October 30, 2012 7:22:18 AM Subject: [vdsm] A question about the SPM operation permission in VDSM
Hi,
In the VDSM code about some SPM operations like HSM.deleteImage(), It is found that VDSM doesn't check if the operation will be launched on a SPM host or not. It only checks if the storage pool is already acquired by one SPM host, but not necessary the same host as the SPM operation is delivered to. The code is like this:
HSM.deleteImage() { ... HSM._spmSchedule() { self.validateSPM(spUUID) <--- Only check if the storage pool was acquired by one host, but not necessary this host } ... }
So it really depends on the node management application AKA ovirt-engine to dispatch the SPM operations to the right VDSM host.
Hi Shu, validateSPM is:
def validateSPM(self, spUUID): pool = self.getPool(spUUID) if pool.spmRole != sp.SPM_ACQUIRED: raise se.SpmStatusError(spUUID)
despite its ambiguous name "SPM_ACQUIRED" refers only to the spmRole of the current host. That said, vdsm actually checks before running deleteImage if the host is actually the SPM or not. Eventually you can verify it running deleteImage on an HSM host, it should fail with:
# vdsClient 0 deleteImage ... Not SPM: ('<spUUID>',)
On 01/31/2013 03:55 PM, Federico Simoncelli wrote:
----- Original Message -----
From: "Shu Ming" shuming@linux.vnet.ibm.com To: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Tuesday, October 30, 2012 7:22:18 AM Subject: [vdsm] A question about the SPM operation permission in VDSM
Hi,
In the VDSM code about some SPM operations like HSM.deleteImage(), It is found that VDSM doesn't check if the operation will be launched on a SPM host or not. It only checks if the storage pool is already acquired by one SPM host, but not necessary the same host as the SPM operation is delivered to. The code is like this:
HSM.deleteImage() { ... HSM._spmSchedule() { self.validateSPM(spUUID) <--- Only check if the storage pool was acquired by one host, but not necessary this host } ... }
So it really depends on the node management application AKA ovirt-engine to dispatch the SPM operations to the right VDSM host.
Hi Shu, validateSPM is:
def validateSPM(self, spUUID): pool = self.getPool(spUUID) if pool.spmRole != sp.SPM_ACQUIRED: raise se.SpmStatusError(spUUID)
despite its ambiguous name "SPM_ACQUIRED" refers only to the spmRole of the current host. That said, vdsm actually checks before running deleteImage if the host is actually the SPM or not. Eventually you can verify it running deleteImage on an HSM host, it should fail with:
# vdsClient 0 deleteImage ... Not SPM: ('<spUUID>',)
VDSM doesn't block API functions at all, it allows you to perform any api functions in any host whether if its SPM or not. What seems wrong, due to that SPM operations are allowed only if VDSM internal state is set to SPM, so why we publish code that we don't use.. seems like it was changed during the time (the dispatcher should hide from the API functions that only the SPM can perform). Now we have the validation as part of sp.py (pool operations), that uses securable.py for that. little bit complicated... but this is how it currently works..
vdsm-devel@lists.fedorahosted.org