| ----- Original Message ----- | | From: "Assaf Muller" amuller@redhat.com | | To: "Michal Skrivanek" michal.skrivanek@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org | | Sent: Thursday, May 23, 2013 2:12:47 PM | | Subject: Re: [vdsm] Migration regression on master | | | | As you can see in a previous patch set I checked if the alias attribute | | exists instead of assuming it exists. | | I then changed my mind with Dan's blessing, and decided to assume it does | | exist, exactly for cases like this. | | Even if we check if the alias exists, what do we do if we find out it | | doesn't? We're at a problem and need to understand why the alias doesn't | | exist because it should - For all devices. | | | | We definitely need to deal with this issue - Can you provide the domxml of | | the VM during creation, and during migration?
Hi Assaf, Peter today has reproduced the same error and provided me the output log, you can find it here: http://etherpad.ovirt.org/p/migration-errors a previous patch blocked by this issue has been accepted (http://gerrit.ovirt.org/#/c/14203/), I've been able to do a migration using the engine. Feel free to ask if you need more details, I will look into this issue in the next few days.
Giuseppe
| | | | | ----- Original Message ----- | | From: "Michal Skrivanek" michal.skrivanek@redhat.com | | To: "Vinzenz Feenstra" vfeenstr@redhat.com, "Assaf Muller" | | amuller@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org, "Dan Kenigsberg" danken@redhat.com | | Sent: Wednesday, May 22, 2013 3:56:22 PM | | Subject: Re: [vdsm] Migration regression on master | | | | | | On May 22, 2013, at 10:14 , Vinzenz Feenstra vfeenstr@redhat.com wrote: | | | | > Hi, | | > | | > During the verification of the refactoring in VDSM (libvirtvm + vm == vm) | | > we have encountered some issue. | | > There seems to be a regression on master branch regarding device aliases. | | > | | > On the target machine _updateDevicesDomxmlCache accesses dev.alias | | > without | | > the attribute being present. | | > | | > This happens to happen at least on balloon devices. | | > I am not sure why we do have this, however it is reproducible and I am | | > not | | > sure why we do this now. | | > | | > I know it has been introduced by http://gerrit.ovirt.org/#/c/13876 and | | > it's | | > actually in the commit message that we'd fail. | | :-) you mean: | | * We now crash horribly if an alias was not found for a device | | when updating devices domxml cache after migrating | | | | > Which is the case... So can somebody please enlighten me and tell me if | | > this is a configuration issue of the VM or if this really a regression. | | not all devices have alias so I think this is indeed a regression | | | | | | > It's possible that the tests aren't properly configured, however that | | > would | | > from my understanding mean that we potentially will run into troubles | | > during migrations from 3.2 -> 3.3 vdsm. | | > | | > PS: | | > This kind of blocks a bit the refactoring verification :/ | | > | | > -- | | > Regards, | | > | | > Vinzenz Feenstra | Senior Software Engineer | | > RedHat Engineering Virtualization R & D | | > Phone: +420 532 294 625 | | > IRC: vfeenstr or evilissimo | | > | | > Better technology. Faster innovation. Powered by community collaboration. | | > See how it works at redhat.com | | > | | > _______________________________________________ | | > vdsm-devel mailing list | | > vdsm-devel@lists.fedorahosted.org | | > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel | | | | _______________________________________________ | | vdsm-devel mailing list | | vdsm-devel@lists.fedorahosted.org | | https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel | | |
On Tue, May 28, 2013 at 11:54:45AM -0400, Giuseppe Vallarelli wrote:
| ----- Original Message ----- | | From: "Assaf Muller" amuller@redhat.com | | To: "Michal Skrivanek" michal.skrivanek@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org | | Sent: Thursday, May 23, 2013 2:12:47 PM | | Subject: Re: [vdsm] Migration regression on master | | | | As you can see in a previous patch set I checked if the alias attribute | | exists instead of assuming it exists. | | I then changed my mind with Dan's blessing, and decided to assume it does | | exist, exactly for cases like this. | | Even if we check if the alias exists, what do we do if we find out it | | doesn't? We're at a problem and need to understand why the alias doesn't | | exist because it should - For all devices. | | | | We definitely need to deal with this issue - Can you provide the domxml of | | the VM during creation, and during migration?
Hi Assaf, Peter today has reproduced the same error and provided me the output log, you can find it here: http://etherpad.ovirt.org/p/migration-errors
Would you add the original vmCreate line, and domxml, that was used to create the VM at the source host?
a previous patch blocked by this issue has been accepted (http://gerrit.ovirt.org/#/c/14203/), I've been able to do a migration using the engine. Feel free to ask if you need more details, I will look into this issue in the next few days.
Giuseppe
On 05/29/2013 07:17 AM, Dan Kenigsberg wrote:
On Tue, May 28, 2013 at 11:54:45AM -0400, Giuseppe Vallarelli wrote:
| ----- Original Message ----- | | From: "Assaf Muller" amuller@redhat.com | | To: "Michal Skrivanek" michal.skrivanek@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org | | Sent: Thursday, May 23, 2013 2:12:47 PM | | Subject: Re: [vdsm] Migration regression on master | | | | As you can see in a previous patch set I checked if the alias attribute | | exists instead of assuming it exists. | | I then changed my mind with Dan's blessing, and decided to assume it does | | exist, exactly for cases like this. | | Even if we check if the alias exists, what do we do if we find out it | | doesn't? We're at a problem and need to understand why the alias doesn't | | exist because it should - For all devices. | | | | We definitely need to deal with this issue - Can you provide the domxml of | | the VM during creation, and during migration?
Hi Assaf, Peter today has reproduced the same error and provided me the output log, you can find it here: http://etherpad.ovirt.org/p/migration-errors
Would you add the original vmCreate line, and domxml, that was used to create the VM at the source host?
http://pastebin.test.redhat.com/144447 http://pastebin.test.redhat.com/144446
On Wed, May 29, 2013 at 02:13:02PM +0200, Peter V. Saveliev wrote:
On 05/29/2013 07:17 AM, Dan Kenigsberg wrote:
On Tue, May 28, 2013 at 11:54:45AM -0400, Giuseppe Vallarelli wrote:
| ----- Original Message ----- | | From: "Assaf Muller" amuller@redhat.com | | To: "Michal Skrivanek" michal.skrivanek@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org | | Sent: Thursday, May 23, 2013 2:12:47 PM | | Subject: Re: [vdsm] Migration regression on master | | | | As you can see in a previous patch set I checked if the alias attribute | | exists instead of assuming it exists. | | I then changed my mind with Dan's blessing, and decided to assume it does | | exist, exactly for cases like this. | | Even if we check if the alias exists, what do we do if we find out it | | doesn't? We're at a problem and need to understand why the alias doesn't | | exist because it should - For all devices. | | | | We definitely need to deal with this issue - Can you provide the domxml of | | the VM during creation, and during migration?
Hi Assaf, Peter today has reproduced the same error and provided me the output log, you can find it here: http://etherpad.ovirt.org/p/migration-errors
Would you add the original vmCreate line, and domxml, that was used to create the VM at the source host?
http://pastebin.test.redhat.com/144447 http://pastebin.test.redhat.com/144446
That's an internal pastebin instance. Could you simply inline the answer? Hyperlinks makes a casual reader of the mailing list not understand what's going on.
Note that we need vmCreate command, the response is less interesting, thow seeing the whole vdsm thread would not harm.
Dan.
* We still need to see the vmCreate command used to learn more about the issue * We can always implement a hack that checks if the alias attribute exists. However, there is probably a deeper issue at hand and this is a good opportunity to find and solve it * Dan has a theory that the issue is with migrating a VM from a 3.0 VDSM version host to a 3.2 host. Perhaps the balloon device isn't created properly at the source host
From: "Dan Kenigsberg" danken@redhat.com To: "Peter V. Saveliev" peet@redhat.com Cc: vdsm-devel@lists.fedorahosted.org Sent: Wednesday, May 29, 2013 4:41:24 PM Subject: Re: [vdsm] Migration regression on master
On Wed, May 29, 2013 at 02:13:02PM +0200, Peter V. Saveliev wrote:
On 05/29/2013 07:17 AM, Dan Kenigsberg wrote:
On Tue, May 28, 2013 at 11:54:45AM -0400, Giuseppe Vallarelli wrote:
| ----- Original Message -----
| | From: "Assaf Muller" amuller@redhat.com | | To: "Michal Skrivanek" michal.skrivanek@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org | | Sent: Thursday, May 23, 2013 2:12:47 PM | | Subject: Re: [vdsm] Migration regression on master | | | | As you can see in a previous patch set I checked if the alias attribute | | exists instead of assuming it exists. | | I then changed my mind with Dan's blessing, and decided to assume it does | | exist, exactly for cases like this. | | Even if we check if the alias exists, what do we do if we find out it | | doesn't? We're at a problem and need to understand why the alias doesn't | | exist because it should - For all devices. | | | | We definitely need to deal with this issue - Can you provide the domxml of | | the VM during creation, and during migration?
Hi Assaf, Peter today has reproduced the same error and provided me the output log, you can find it here: http://etherpad.ovirt.org/p/migration-errors
Would you add the original vmCreate line, and domxml, that was used to create the VM at the source host?
http://pastebin.test.redhat.com/144447 http://pastebin.test.redhat.com/144446
That's an internal pastebin instance. Could you simply inline the answer? Hyperlinks makes a casual reader of the mailing list not understand what's going on.
Note that we need vmCreate command, the response is less interesting, thow seeing the whole vdsm thread would not harm.
Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
Interesting Theory. IIRC in the past the balloon device was auto created by libvirt, and at some point vdsm started removing it. So if the migration is from an old version where the device is created to a new one which supports a balloon created by the engine, it may be missing some parameters. So indeed we need to understand more about the source vdsm and how its VMs were created.
----- Original Message -----
From: "Assaf Muller" amuller@redhat.com To: "Dan Kenigsberg" danken@redhat.com Cc: vdsm-devel@lists.fedorahosted.org Sent: Thursday, May 30, 2013 10:00:49 AM Subject: Re: [vdsm] Migration regression on master
* We still need to see the vmCreate command used to learn more about the issue * We can always implement a hack that checks if the alias attribute exists. However, there is probably a deeper issue at hand and this is a good opportunity to find and solve it * Dan has a theory that the issue is with migrating a VM from a 3.0 VDSM version host to a 3.2 host. Perhaps the balloon device isn't created properly at the source host
From: "Dan Kenigsberg" danken@redhat.com To: "Peter V. Saveliev" peet@redhat.com Cc: vdsm-devel@lists.fedorahosted.org Sent: Wednesday, May 29, 2013 4:41:24 PM Subject: Re: [vdsm] Migration regression on master
On Wed, May 29, 2013 at 02:13:02PM +0200, Peter V. Saveliev wrote:
On 05/29/2013 07:17 AM, Dan Kenigsberg wrote:
On Tue, May 28, 2013 at 11:54:45AM -0400, Giuseppe Vallarelli wrote:
| ----- Original Message -----
| | From: "Assaf Muller" amuller@redhat.com | | To: "Michal Skrivanek" michal.skrivanek@redhat.com | | Cc: "vdsm-devel@lists.fedorahosted.org Development" | | vdsm-devel@lists.fedorahosted.org | | Sent: Thursday, May 23, 2013 2:12:47 PM | | Subject: Re: [vdsm] Migration regression on master | | | | As you can see in a previous patch set I checked if the alias | | attribute | | exists instead of assuming it exists. | | I then changed my mind with Dan's blessing, and decided to assume it | | does | | exist, exactly for cases like this. | | Even if we check if the alias exists, what do we do if we find out it | | doesn't? We're at a problem and need to understand why the alias | | doesn't | | exist because it should - For all devices. | | | | We definitely need to deal with this issue - Can you provide the | | domxml of | | the VM during creation, and during migration?
Hi Assaf, Peter today has reproduced the same error and provided me the output log, you can find it here: http://etherpad.ovirt.org/p/migration-errors
Would you add the original vmCreate line, and domxml, that was used to create the VM at the source host?
http://pastebin.test.redhat.com/144447 http://pastebin.test.redhat.com/144446
That's an internal pastebin instance. Could you simply inline the answer? Hyperlinks makes a casual reader of the mailing list not understand what's going on.
Note that we need vmCreate command, the response is less interesting, thow seeing the whole vdsm thread would not harm.
Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
----- Original Message ----- | From: "Assaf Muller" amuller@redhat.com | To: "Dan Kenigsberg" danken@redhat.com | Cc: "Peter V. Saveliev" peet@redhat.com, vdsm-devel@lists.fedorahosted.org, "Giuseppe Vallarelli" | gvallare@redhat.com, "michal skrivanek" michal.skrivanek@redhat.com | Sent: Thursday, May 30, 2013 9:00:49 AM | Subject: Re: [vdsm] Migration regression on master | | | | * We still need to see the vmCreate command used to learn more about the | issue | * We can always implement a hack that checks if the alias attribute | exists. However, there is probably a deeper issue at hand and this is a | good opportunity to find and solve it | * Dan has a theory that the issue is with migrating a VM from a 3.0 VDSM | version host to a 3.2 host. Perhaps the balloon device isn't created | properly at the source host
I've been able to reproduce the failure here there are more details:
On the source host
* vmCreate http://pastebin.com/yEZ3zkyN * vm xml http://pastebin.com/YSy4TYca
On the dest host
* vmMigrationCreate http://pastebin.com/rwdPC8Ef * _srcDomXml nicely formatted http://pastebin.com/pjE8sMMW
Giuseppe
| | From: "Dan Kenigsberg" danken@redhat.com | To: "Peter V. Saveliev" peet@redhat.com | Cc: vdsm-devel@lists.fedorahosted.org | Sent: Wednesday, May 29, 2013 4:41:24 PM | Subject: Re: [vdsm] Migration regression on master | | On Wed, May 29, 2013 at 02:13:02PM +0200, Peter V. Saveliev wrote: | > On 05/29/2013 07:17 AM, Dan Kenigsberg wrote: | > >On Tue, May 28, 2013 at 11:54:45AM -0400, Giuseppe Vallarelli wrote: | > >> | ----- Original Message ----- | | > >>| | From: "Assaf Muller" amuller@redhat.com | > >>| | To: "Michal Skrivanek" michal.skrivanek@redhat.com | > >>| | Cc: "vdsm-devel@lists.fedorahosted.org Development" | > >>| | vdsm-devel@lists.fedorahosted.org | > >>| | Sent: Thursday, May 23, 2013 2:12:47 PM | > >>| | Subject: Re: [vdsm] Migration regression on master | > >>| | | > >>| | As you can see in a previous patch set I checked if the alias | > >>| | attribute | > >>| | exists instead of assuming it exists. | > >>| | I then changed my mind with Dan's blessing, and decided to assume it | > >>| | does | > >>| | exist, exactly for cases like this. | > >>| | Even if we check if the alias exists, what do we do if we find out it | > >>| | doesn't? We're at a problem and need to understand why the alias | > >>| | doesn't | > >>| | exist because it should - For all devices. | > >>| | | > >>| | We definitely need to deal with this issue - Can you provide the | > >>| | domxml of | > >>| | the VM during creation, and during migration? | > >> | > >>Hi Assaf, Peter today has reproduced the same error and provided me | > >>the output log, you can find it here: | > >>http://etherpad.ovirt.org/p/migration-errors | > > | > >Would you add the original vmCreate line, and domxml, that was used to | > >create the VM at the source host? | > | > http://pastebin.test.redhat.com/144447 | > http://pastebin.test.redhat.com/144446 | | That's an internal pastebin instance. Could you simply inline the | answer? Hyperlinks makes a casual reader of the mailing list not | understand what's going on. | | Note that we need vmCreate command, the response is less | interesting, thow seeing the whole vdsm thread would not harm. | | Dan. | _______________________________________________ | vdsm-devel mailing list | vdsm-devel@lists.fedorahosted.org | https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel | |
vdsm-devel@lists.fedorahosted.org