Change in vdsm[master]: Live Merge: work around racy libvirt pivot

alitke at redhat.com alitke at redhat.com
Fri Mar 27 21:33:33 UTC 2015


Adam Litke has uploaded a new change for review.

Change subject: Live Merge: work around racy libvirt pivot
......................................................................

Live Merge: work around racy libvirt pivot

Libvirt version 1.2.8-16.el7_1.2 introduced a bug where the synchronous
call to blockJobAbort (which we use to pivot to the new leaf after an
active layer merge completes) will return before the domain XML has been
updated.  This makes it look like the pivot failed when it actually
succeeded.  This means that vdsm state will not be properly synchronized
and we may start the vm with a stale volume in the future which will
corrupt the VM's storage.  See
https://bugzilla.redhat.com/show_bug.cgi?id=1202719 for more details.

Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=1206722
Change-Id: I1e794622baf66c75cbe583be03a7b9a4a7e4883d
Signed-off-by: Adam Litke <alitke at redhat.com>
---
M vdsm/virt/vm.py
1 file changed, 28 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/03/39303/1

diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py
index 28054bf..bafe508 100644
--- a/vdsm/virt/vm.py
+++ b/vdsm/virt/vm.py
@@ -5061,6 +5061,33 @@
         self.doPivot = doPivot
         self.success = False
 
+    def _waitForBuggyLibvirt(self):
+        # Libvirt version 1.2.8-16.el7_1.2 introduced a bug where the
+        # synchronous call to blockJobAbort will return before the domain XML
+        # has been updated.  This makes it look like the pivot failed when it
+        # actually succeeded.  This means that vdsm state will not be properly
+        # synchronized and we may start the vm with a stale volume in the
+        # future.  See https://bugzilla.redhat.com/show_bug.cgi?id=1202719 for
+        # more details.
+        #
+        # TODO: Remove once we depend on a libvirt with this bug fixed.
+        origVols = [x['volumeID'] for x in self.drive.volumeChain]
+        alias = self.drive['alias']
+        while True:
+            chains = self.vm._driveGetActualVolumeChain([self.drive])
+            try:
+                curVols = chains[alias]
+            except KeyError:
+                raise RuntimeError("Failed to retrieve volume chain for "
+                                   "drive:%s.  Pivot failed.")
+            if len(curVols) < len(origVols):
+                # The merged volume has now been removed from the chain
+                break
+
+            self.vm.log.debug("Waiting for libvirt to update the XML after "
+                              "pivot of drive %s completed", alias)
+            time.sleep(1)
+
     def tryPivot(self):
         # We call imageSyncVolumeChain which will mark the current leaf
         # ILLEGAL.  We do this before requesting a pivot so that we can
@@ -5092,6 +5119,7 @@
                 self.vm.log.error("Pivot failed for job %s (rc=%i)",
                                   self.jobId, ret)
                 raise RuntimeError("pivot failed")
+            self._waitForBuggyLibvirt()
         self.vm.log.info("Pivot completed (job %s)", self.jobId)
 
     @utils.traceback()


-- 
To view, visit https://gerrit.ovirt.org/39303
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1e794622baf66c75cbe583be03a7b9a4a7e4883d
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Adam Litke <alitke at redhat.com>


More information about the vdsm-patches mailing list