Milan Zamazal has uploaded a new change for review.
Change subject: virt: Destroy VM after post-copy migration failure ......................................................................
virt: Destroy VM after post-copy migration failure
As explained in the source code comment, we don't have currently a better option than to destroy the VM remnants after a failed post-copy migration. This may change in future, if a failed post-copy migration recovery is available in libvirt/QEMU.
Change-Id: I1918e9afce189c8b3f617766e55afa13f1e153f1 Signed-off-by: Milan Zamazal mzamazal@redhat.com Bug-Url: https://bugzilla.redhat.com/1354343 --- M lib/vdsm/virt/vmexitreason.py M vdsm/virt/vm.py 2 files changed, 24 insertions(+), 1 deletion(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/42/64142/7
diff --git a/lib/vdsm/virt/vmexitreason.py b/lib/vdsm/virt/vmexitreason.py index 46c092b..494cd28 100644 --- a/lib/vdsm/virt/vmexitreason.py +++ b/lib/vdsm/virt/vmexitreason.py @@ -30,6 +30,7 @@ MIGRATION_FAILED = 8 LIBVIRT_DOMAIN_MISSING = 9 DESTROYED_ON_STARTUP = 10 +POSTCOPY_MIGRATION_FAILED = 11
exitReasons = { @@ -44,4 +45,5 @@ MIGRATION_FAILED: 'VM failed to migrate', LIBVIRT_DOMAIN_MISSING: 'Failed to find the libvirt domain', DESTROYED_ON_STARTUP: 'VM destroyed during the startup', + POSTCOPY_MIGRATION_FAILED: 'Migration failed in post-copy', } diff --git a/vdsm/virt/vm.py b/vdsm/virt/vm.py index ed60354..e274388 100644 --- a/vdsm/virt/vm.py +++ b/vdsm/virt/vm.py @@ -4163,7 +4163,28 @@ else: hooks.after_vm_pause(domxml, self.conf) elif detail == libvirt.VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED: - pass # will be handled in a followup patch + # This can happen on both the ends of the migration. + # After a failed post-copy migration, the VM remains in a + # paused state on both the ends of the migration. There is + # currently no way to recover it, since the VM is missing some + # memory pages on the destination and the old snapshot at the + # source doesn't know about the changes made to the external + # world (network, storage, ...) during the post-copy phase. + # The best what we can do in such a situation is to destroy the + # paused VM instances on both the ends before someone tries to + # resume any of them, causing confusion at best or more damages + # in the worse case. We must also inform Engine about the + # fatal state of the failed migration, so we can't destroy the + # VM immediately on the destination (but we can do it on the + # source). We report the VM as down on the destination to + # Engine and wait for destroy request from it. + self.log.warning("Migration failed in post-copy, " + "destroying VM: %s" % (self.id,)) + destroy = self.lastStatus == vmstatus.MIGRATION_SOURCE + self.setDownStatus(ERROR, + vmexitreason.POSTCOPY_MIGRATION_FAILED) + if destroy: + self.destroy()
elif event == libvirt.VIR_DOMAIN_EVENT_RESUMED: self._setGuestCpuRunning(True)
gerrit-hooks has posted comments on this change.
Change subject: virt: Destroy VM after failed post-copy migration ......................................................................
Patch Set 1:
* Update tracker: IGNORE, no Bug-Url found * Check Bug-Url::WARN, no bug url found, make sure header matches 'Bug-Url: ' and is a valid url. * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-4.0'])
Francesco Romani has posted comments on this change.
Change subject: virt: Destroy VM after post-copy migration failure ......................................................................
Patch Set 7: Code-Review+2
gerrit-hooks has posted comments on this change.
Change subject: virt: Destroy VM after post-copy migration failure ......................................................................
Patch Set 8:
* Update Tracker::#1354343::OK, status: POST * Check Bug-Url::IGNORE, not relevant for branch: master * Check Public Bug::#1354343::OK, public bug * Check Product::IGNORE, not relevant for branch: master * Check TM::IGNORE, not relevant for branch: master * Check merged to previous::IGNORE, Not in stable branch (['ovirt-3.6', 'ovirt-4.0'])
Milan Zamazal has posted comments on this change.
Change subject: virt: Destroy VM after post-copy migration failure ......................................................................
Patch Set 8:
Just rebase + resolved conflicts.
Francesco Romani has posted comments on this change.
Change subject: virt: Destroy VM after post-copy migration failure ......................................................................
Patch Set 8: Code-Review+2
From Dan Kenigsberg danken@redhat.com:
Dan Kenigsberg has submitted this change and it was merged.
Change subject: virt: Destroy VM after post-copy migration failure ......................................................................
virt: Destroy VM after post-copy migration failure
As explained in the source code comment, we don't have currently a better option than to destroy the VM remnants after a failed post-copy migration. This may change in future, if a failed post-copy migration recovery is available in libvirt/QEMU.
How do we know that a post-copy migration failed? On the source, the migration simply fails and the VM either remains in a paused status with a post-copy reason or it disappears. On the destination, the VM can end up in the same states, but there are two more things to consider: 1. we may not be aware that we are in post-copy (if the corresponding event is not sent from libvirt before the migration fails); 2. if the VM gets paused, we receive VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED event.
Disappeared VMs are already handled as general QEMU crashes. We must specifically handle the paused case on both the ends, since this is a new situation and we want to destroy the VM immediately. We also delete the VM object on the source, since Engine handles only the destination in post-copy migrations.
Change-Id: I1918e9afce189c8b3f617766e55afa13f1e153f1 Signed-off-by: Milan Zamazal mzamazal@redhat.com Bug-Url: https://bugzilla.redhat.com/1354343 --- M lib/vdsm/virt/migration.py M lib/vdsm/virt/vmexitreason.py M vdsm/virt/vm.py 3 files changed, 32 insertions(+), 1 deletion(-)
Approvals: Jenkins CI: Passed CI tests Francesco Romani: Looks good to me, approved Milan Zamazal: Verified
vdsm-patches@lists.fedorahosted.org