Change in vdsm[ovirt-3.6]: periodic: ignore known-benign libvirt errors

fromani at redhat.com fromani at redhat.com
Wed Feb 24 15:28:12 UTC 2016


Hello Dan Kenigsberg, Milan Zamazal,

I'd like you to do a code review.  Please visit

    https://gerrit.ovirt.org/53999

to review the following change.

Change subject: periodic: ignore known-benign libvirt errors
......................................................................

periodic: ignore known-benign libvirt errors

periodic sampling threads and migration thread are
(and need to be) asynchronous with each other.
This leaves the door open for errors like

libvirtError: Domain not found: no domain with matching uuid
'7ebb5925-e76f-4d2f-a148-7671c313fe84' (v2)

Which are harmless but spam the logs.
We don't want to hide real errors, but we want to
avoid this noise, so this patch makes periodic Operation
ignore some of these errors, the ones which are easier
to detect.

Change-Id: Ifb5618f44870bdca97fa479734514a3c8d81c7d7
Bug-Url: https://bugzilla.redhat.com/1299480
Backport-To: 3.6
Signed-off-by: Francesco Romani <fromani at redhat.com>
Reviewed-on: https://gerrit.ovirt.org/52884
Reviewed-by: Milan Zamazal <mzamazal at redhat.com>
Continuous-Integration: Jenkins CI
Reviewed-by: Dan Kenigsberg <danken at redhat.com>
---
M tests/periodicTests.py
M vdsm/virt/periodic.py
2 files changed, 60 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/99/53999/1

diff --git a/tests/periodicTests.py b/tests/periodicTests.py
index 21a0e4c..3e91a0c 100644
--- a/tests/periodicTests.py
+++ b/tests/periodicTests.py
@@ -22,11 +22,14 @@
 import threading
 import time
 
+import libvirt
+
 from vdsm import executor
 from vdsm import schedule
 from vdsm.utils import monotonic_time
 
 from virt import periodic
+from virt import vmstatus
 
 from testlib import expandPermutations, permutations
 from testlib import VdsmTestCase as TestCaseBase
@@ -233,6 +236,42 @@
                     vm_id, vm_id)
 
 
+ at expandPermutations
+class NumaInfoMonitorTests(TestCaseBase):
+
+    def setUp(self):
+        self.vm_id = _fake_vm_id(0)
+        self.vm = _FakeVM(self.vm_id, self.vm_id)
+        self.op = periodic.NumaInfoMonitor(self.vm)
+
+    @permutations([
+        # errcode, migrating, last_status
+        [libvirt.VIR_ERR_NO_DOMAIN, True, vmstatus.UP],
+        [libvirt.VIR_ERR_NO_DOMAIN, False, vmstatus.DOWN],
+    ])
+    def test_swallow_exceptions(self, errcode, migrating, last_status):
+
+        def fail(*args):
+            raise fake.Error(errcode)
+
+        self.vm.updateNumaInfo = fail
+
+        self.vm.migrating = migrating
+        self.vm.lastStatus = last_status
+        self.assertNotRaises(self.op)
+
+    def test_propagate_exceptions(self):
+
+        def fail(*args):
+            raise fake.Error(libvirt.VIR_ERR_NO_DOMAIN)
+
+        self.vm.updateNumaInfo = fail
+
+        self.vm.migrating = False
+        self.vm.lastStatus = vmstatus.UP
+        self.assertRaises(libvirt.libvirtError, self.op)
+
+
 def _fake_vm_id(i):
     return 'VM-%03i' % i
 
@@ -290,3 +329,11 @@
     def __init__(self, vmId, vmName):
         self.id = vmId
         self.name = vmName
+        self.migrating = False
+        self.lastStatus = vmstatus.UP
+
+    def isMigrating(self):
+        return self.migrating
+
+    def updateNumaInfo(self):
+        pass
diff --git a/vdsm/virt/periodic.py b/vdsm/virt/periodic.py
index 4d64522..509b82d 100644
--- a/vdsm/virt/periodic.py
+++ b/vdsm/virt/periodic.py
@@ -25,12 +25,15 @@
 import logging
 import threading
 
+import libvirt
+
 from vdsm import executor
 from vdsm import libvirtconnection
 from vdsm.config import config
 
 from . import sampling
 from . import virdomain
+from . import vmstatus
 
 
 # just a made up number. Maybe should be equal to number of cores?
@@ -275,6 +278,7 @@
         return self._vm.isDomainReadyForCommands()
 
     def __call__(self):
+        migrating = self._vm.isMigrating()
         try:
             self._execute()
         except virdomain.NotConnectedError:
@@ -283,6 +287,15 @@
             # both cases: let's reduce the log spam.
             self._vm.log.warning('could not run on %s: domain not connected',
                                  self._vm.id)
+        except libvirt.libvirtError as e:
+            if e.get_error_code() in (
+                # race on shutdown/migration completion
+                libvirt.VIR_ERR_NO_DOMAIN,
+            ):
+                # known benign cases: migration in progress or completed
+                if migrating or self._vm.lastStatus == vmstatus.DOWN:
+                    return
+            raise
 
     def _execute(self):
         raise NotImplementedError


-- 
To view, visit https://gerrit.ovirt.org/53999
To unsubscribe, visit https://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifb5618f44870bdca97fa479734514a3c8d81c7d7
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: ovirt-3.6
Gerrit-Owner: Francesco Romani <fromani at redhat.com>
Gerrit-Reviewer: Dan Kenigsberg <danken at redhat.com>
Gerrit-Reviewer: Milan Zamazal <mzamazal at redhat.com>


More information about the vdsm-patches mailing list