Hi,
I am testing the VM migration in oVirt engine 3.1 bteta release and successfully did the migration test on NFS date center. However, the test of VM migration on iSCSI data center failed. I dumped the error and warning segments from the vdsm logs on both the source and destination host. Please ignore the time stamp out of sync between the two hosts. _ _*The vdsm log segment from the source host:*
66-4da5-8310-6dcc970e5367`::migration downtime thread exiting Thread-115166::ERROR::2012-07-25 23:18:55,953::vm::176::vm.Vm::(_recover) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::operation failed: Failed to connect to remote libvirt URI qemu+tcp://9.181.129.110/system Dummy-115148::DEBUG::2012-07-25 23:18:55,981::__init__::1249::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/0b9a4ea4-d487-11e1-b614-5254001498c4/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) Thread-115166::ERROR::2012-07-25 23:18:56,028::vm::240::vm.Vm::(run) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 223, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 451, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/libvirtvm.py", line 491, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1034, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://9.181.129.110/system
*The vdsm log segment from the destination host:*
Thread-1577::INFO::2012-07-25 23:19:50,407::API::601::vds::(_getNetworkIp) network None: using 0 Thread-1577::INFO::2012-07-25 23:19:50,407::API::228::vds::(create) vmContainerLock acquired by vm 11e06222-2d66-4da5-8310-6dcc970e5367 Thread-1578::DEBUG::2012-07-25 23:19:50,411::vm::564::vm.Vm::(_startUnderlyingVm) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Start Thread-1577::DEBUG::2012-07-25 23:19:50,411::API::244::vds::(create) Total desktops after creation of 11e06222-2d66-4da5-8310-6dcc970e5367 is 1 Thread-1578::DEBUG::2012-07-25 23:19:50,411::vm::568::vm.Vm::(_startUnderlyingVm) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::_ongoingCreations acquired Thread-1577::DEBUG::2012-07-25 23:19:50,412::libvirtvm::2438::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::migration destination: waiting 36s for path preparation Thread-1578::INFO::2012-07-25 23:19:50,412::libvirtvm::1285::vm.Vm::(_run) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::VM wrapper has started Thread-1578::WARNING::2012-07-25 23:19:50,413::vm::398::vm.Vm::(getConfDevices) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Unknown type found, device: '{'device': 'unix', 'alias': 'channel0', 'type': 'channel', 'address': {'bus': '0', 'controller': '0', 'type': 'virtio-serial', 'port': '1'}}' found
Thread-1578::ERROR::2012-07-25 23:24:50,484::vm::604::vm.Vm::(_startUnderlyingVm) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::The vm start process failed Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 584, in _startUnderlyingVm self._waitForIncomingMigrationFinish() File "/usr/share/vdsm/libvirtvm.py", line 1572, in _waitForIncomingMigrationFinish self._connection.lookupByUUIDString(self.id), File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2608, in lookupByUUIDString if ret is None:raise libvirtError('virDomainLookupByUUIDString() failed', conn=self) libvirtError: Domain not found: no domain with matching uuid '11e06222-2d66-4da5-8310-6dcc970e5367' - Timed out (did not receive success event) Thread-1578::DEBUG::2012-07-25 23:24:50,485::vm::920::vm.Vm::(setDownStatus) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Changed state to Down: Domain not found: no domain with matching uuid '11e0622
Any idea?
----- Original Message -----
From: "Shu Ming" shuming@linux.vnet.ibm.com To: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org Sent: Wednesday, July 25, 2012 5:57:05 PM Subject: [vdsm] Fatal error when migrate the VM with disk on iscsi data center
Hi,
I am testing the VM migration in oVirt engine 3.1 bteta release and successfully did the migration test on NFS date center. However, the test of VM migration on iSCSI data center failed. I dumped the error and warning segments from the vdsm logs on both the source and destination host. Please ignore the time stamp out of sync between the two hosts.
The vdsm log segment from the source host:
66-4da5-8310-6dcc970e5367`::migration downtime thread exiting Thread-115166::ERROR::2012-07-25 23:18:55,953::vm::176::vm.Vm::(_recover) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::operation failed: Failed to connect to remote libvirt URI qemu+tcp://9.181.129.110/system Dummy-115148::DEBUG::2012-07-25 23:18:55,981::__init__::1249::Storage.Misc.excCmd::(_log) 'dd if=/rhev/data-center/0b9a4ea4-d487-11e1-b614-5254001498c4/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) Thread-115166::ERROR::2012-07-25 23:18:56,028::vm::240::vm.Vm::(run) vmId=`11e06222-2d66-4da5-8310-6dcc970e5367`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/vm.py", line 223, in run self._startUnderlyingMigration() File "/usr/share/vdsm/libvirtvm.py", line 451, in _startUnderlyingMigration None, maxBandwidth) File "/usr/share/vdsm/libvirtvm.py", line 491, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 82, in wrapper ret = f(*args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1034, in migrateToURI2 if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://9.181.129.110/system
The destination host doesn't look right (9.181.129.110). Can you post the logs related to the vmMigrate parameters? Eventually you could open a bug so that you can attach the full logs.
vdsm-devel@lists.fedorahosted.org