Change in vdsm[ovirt-3.5]: Live Merge: work around racy libvirt pivot

alitke at redhat.com alitke at redhat.com
Tue Mar 31 18:43:47 UTC 2015


Adam Litke has uploaded a new change for review.

Change subject: Live Merge: work around racy libvirt pivot
......................................................................

Live Merge: work around racy libvirt pivot

Libvirt version 1.2.8-16.el7_1.2 introduced a bug where the synchronous
call to blockJobAbort (which we use to pivot to the new leaf after an
active layer merge completes) will return before the domain XML has been
updated.  This makes it look like the pivot failed when it actually
succeeded.  This means that vdsm state will not be properly synchronized
and we may start the vm with a stale volume in the future which will
corrupt the VM's storage.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1202719 for more details.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1207808
Change-Id: I1e794622baf66c75cbe583be03a7b9a4a7e4883d
Signed-off-by: Adam Litke <alitke at redhat.com>
---
M vdsm/virt/vm.py
1 file changed, 44 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/19/39419/1

diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 904c8e1..fe6bf6b 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -6098,6 +6098,7 @@
                 self.vm.log.error("Pivot failed for job %s (rc=%i)",
                                   self.jobId, ret)
                 raise RuntimeError("pivot failed")
+            self._waitForXMLUpdate()
         self.vm.log.info("Pivot completed (job %s)", self.jobId)
 
     @utils.traceback()
@@ -6118,6 +6119,49 @@
         """
         return self.success
 
+    def _waitForXMLUpdate(self):
+        # Libvirt version 1.2.8-16.el7_1.2 introduced a bug where the
+        # synchronous call to blockJobAbort will return before the domain XML
+        # has been updated.  This makes it look like the pivot failed when it
+        # actually succeeded.  This means that vdsm state will not be properly
+        # synchronized and we may start the vm with a stale volume in the
+        # future.  See https://bugzilla.redhat.com/show_bug.cgi?id=1202719 for
+        # more details.
+        # TODO: Remove once we depend on a libvirt with this bug fixed.
+
+        # We expect libvirt to show that the original leaf has been removed
+        # from the active volume chain.
+        origVols = sorted([x['volumeID'] for x in self.drive.volumeChain])
+        expectedVols = origVols[:]
+        expectedVols.remove(self.drive.volumeID)
+
+        alias = self.drive['alias']
+        self.vm.log.info("Waiting for libvirt to update the XML after pivot "
+                         "of drive %s completed", alias)
+        while True:
+            # This operation should complete in either one or two iterations of
+            # this loop.  Until libvirt updates the XML there is nothing to do
+            # but wait.  While we wait we continue to tell engine that the job
+            # is ongoing.  If we are still in this loop when the VM is powered
+            # off, the merge will be resolved manually by engine using the
+            # reconcileVolumeChain verb.
+            chains = self.vm._driveGetActualVolumeChain([self.drive])
+            if alias not in chains.keys():
+                raise RuntimeError("Failed to retrieve volume chain for "
+                                   "drive %s.  Pivot failed.", alias)
+            curVols = sorted([entry.uuid for entry in chains[alias]])
+
+            if curVols == origVols:
+                time.sleep(1)
+            elif curVols == expectedVols:
+                self.vm.log.info("The XML update has been completed")
+                break
+            else:
+                self.log.error("Bad volume chain found for drive %s. Previous "
+                               "chain: %s, Expected chain: %s, Actual chain: "
+                               "%s", alias, origVols, expectedVols, curVols)
+                raise RuntimeError("Bad volume chain found")
+
 
 def _getNetworkIp(network):
     try:


-- 
To view, visit https://gerrit.ovirt.org/39419
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1e794622baf66c75cbe583be03a7b9a4a7e4883d
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.5
Gerrit-Owner: Adam Litke <alitke at redhat.com>


More information about the vdsm-patches mailing list