If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata: ------------------------------------------ The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for? Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
New VDSM APIs required: ----------------------- Once we have a place to store persistent VM configurations, we will need APIs to define and undefine VMs. I propose the following:
define(vm_params): Create a persistent VM. This API will create a persistent VM that will start in the 'Down' state. The initial user parameters will be saved along with any VDSM derived parameters to a metadata file.
undefine(vmUUID): Delete a persistent VM. This API will un-persist a VM that has been previously created with 'define'. This operation is valid only for Down VMs. The VM will be destroyed and its persistent config removed.
Future APIs: In the future, a new class of APIs for modifying the state of an existing VM will be required. For example: add/remove disks, change memory, change cpus/topology, etc. It would not be very user friendly to force redefinition of a VM just to change some simple hardware details.
Other implications: ------------------- Adding this feature will have implications on a number of other APIs. I am sure I am missing many here so please feel free to add information if you know it.
create(): - If passed in a vmId corresponding to a persistent VM, parameters are loaded from stored metadata.
list(), listNames(), getAllVmStats(), getVmsList(): - These functions must also list persistent VMs
migrate(): - After successful migration, remove local VM metadata
Comments? What have I missed?
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
That sub-package could implement the define/undefine API that you suggest; maybe something more elaborated such as rhevm-api.
Regards,
Dan.
New VDSM APIs required:
Once we have a place to store persistent VM configurations, we will need APIs to define and undefine VMs. I propose the following:
define(vm_params): Create a persistent VM. This API will create a persistent VM that will start in the 'Down' state. The initial user parameters will be saved along with any VDSM derived parameters to a metadata file.
undefine(vmUUID): Delete a persistent VM. This API will un-persist a VM that has been previously created with 'define'. This operation is valid only for Down VMs. The VM will be destroyed and its persistent config removed.
Future APIs: In the future, a new class of APIs for modifying the state of an existing VM will be required. For example: add/remove disks, change memory, change cpus/topology, etc. It would not be very user friendly to force redefinition of a VM just to change some simple hardware details.
Other implications:
Adding this feature will have implications on a number of other APIs. I am sure I am missing many here so please feel free to add information if you know it.
create():
- If passed in a vmId corresponding to a persistent VM, parameters are loaded from stored metadata.
list(), listNames(), getAllVmStats(), getVmsList():
- These functions must also list persistent VMs
migrate():
- After successful migration, remove local VM metadata
Comments? What have I missed?
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
That sub-package could implement the define/undefine API that you suggest; maybe something more elaborated such as rhevm-api.
Regards,
Dan.
New VDSM APIs required:
Once we have a place to store persistent VM configurations, we will need APIs to define and undefine VMs. I propose the following:
define(vm_params): Create a persistent VM. This API will create a persistent VM that will start in the 'Down' state. The initial user parameters will be saved along with any VDSM derived parameters to a metadata file.
undefine(vmUUID): Delete a persistent VM. This API will un-persist a VM that has been previously created with 'define'. This operation is valid only for Down VMs. The VM will be destroyed and its persistent config removed.
Future APIs: In the future, a new class of APIs for modifying the state of an existing VM will be required. For example: add/remove disks, change memory, change cpus/topology, etc. It would not be very user friendly to force redefinition of a VM just to change some simple hardware details.
Other implications:
Adding this feature will have implications on a number of other APIs. I am sure I am missing many here so please feel free to add information if you know it.
create():
- If passed in a vmId corresponding to a persistent VM, parameters
are loaded from stored metadata.
list(), listNames(), getAllVmStats(), getVmsList():
- These functions must also list persistent VMs
migrate():
- After successful migration, remove local VM metadata
Comments? What have I missed?
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
The DB may be a mere directory. But it has to be ONE.
Dan.
----- Original Message -----
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
We would still have an "SDM" which would prevent such a scenario from occuring. Management would have to choose the SDM. I don't see the problem.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
The DB may be a mere directory. But it has to be ONE.
Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon, Sep 05, 2011 at 05:59:28AM -0400, Ayal Baron wrote:
----- Original Message -----
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
We would still have an "SDM" which would prevent such a scenario from occuring. Management would have to choose the SDM. I don't see the problem.
I do not see how SDM helps us here, unless we give one of our SDMs control on our single VM DB. One Vdsm is asked to create VM-A. Another Vdsm is asked to create another VM, which happens to have the same name, but a totally different definition. Who decides which one is created?
Dan.
----- Original Message -----
On Mon, Sep 05, 2011 at 05:59:28AM -0400, Ayal Baron wrote:
----- Original Message -----
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
We would still have an "SDM" which would prevent such a scenario from occuring. Management would have to choose the SDM. I don't see the problem.
I do not see how SDM helps us here, unless we give one of our SDMs control on our single VM DB. One Vdsm is asked to create VM-A. Another Vdsm is asked to create another VM, which happens to have the same name, but a totally different definition. Who decides which one is created?
As I noted above, you would have a VM "DB" per domain, not one global DB. And of course only one vdsm would control a domain (the SDM)
Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon, Sep 05, 2011 at 07:38:30AM -0400, Ayal Baron wrote:
----- Original Message -----
On Mon, Sep 05, 2011 at 05:59:28AM -0400, Ayal Baron wrote:
----- Original Message -----
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote: > If VDSM is to be functional in a stand-alone configuration, > one > of > the first > features that must be added is support for persistent > domains. > Since > there will > not be a management server around to "remember" domain > configurations, there > should be an option to have VDSM store it. Here is my idea > for > how > it could be > implemented. Your comments and advice are greatly > appreciated. > > > Selecting a location to store VM metadata: > ------------------------------------------ > The first design decision that needs to be made is where the > VM > metadata should > be stored. Based on my investigation, there are two > potential > locations: 1) In > a new directory under '/rhev/data-center', or 2) Inside the > Master > Storage > Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/'). > > Option #2 seems the most promising at first but I have a few > concerns. The > '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory > already > exists > today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
> Even though I have some VMs > created I do not see any information here. Also, if the > concept > of > Master > Storage Domain is going away, in the future how will we > select a > SD > in which to > store VM definitions? > > Option #1 may be an acceptable fallback if issues with #2 > cannot > be > resolved. > However, I would prefer that VM metadata be stored on > managed > storage so that it > can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
We would still have an "SDM" which would prevent such a scenario from occuring. Management would have to choose the SDM. I don't see the problem.
I do not see how SDM helps us here, unless we give one of our SDMs control on our single VM DB. One Vdsm is asked to create VM-A. Another Vdsm is asked to create another VM, which happens to have the same name, but a totally different definition. Who decides which one is created?
As I noted above, you would have a VM "DB" per domain, not one global DB. And of course only one vdsm would control a domain (the SDM)
The point that I try to convey is that having more than one VM DB in a migration domain, is wrong.
One SDM may be asked to define VM-A, with disks on its domain. Another SDM may be asked to define another VM named VM-A. We end up defining both, and cannot run them both on the same host.
And to our jewish readers: there's one God in heaven; there should be one DB on earth.
Dan.
I do not see how SDM helps us here, unless we give one of our SDMs control on our single VM DB. One Vdsm is asked to create VM-A. Another Vdsm is asked to create another VM, which happens to have the same name, but a totally different definition. Who decides which one is created?
As I noted above, you would have a VM "DB" per domain, not one global DB. And of course only one vdsm would control a domain (the SDM)
The point that I try to convey is that having more than one VM DB in a migration domain, is wrong.
One SDM may be asked to define VM-A, with disks on its domain. Another SDM may be asked to define another VM named VM-A. We end up defining both, and cannot run them both on the same host.
A. Why not? B. that's the user's problem. But it wouldn't cause any corruption.
And to our jewish readers: there's one God in heaven; there should be one DB on earth.
Dan. _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon, Sep 05, 2011 at 01:53:51PM +0300, Dan Kenigsberg wrote:
On Mon, Sep 05, 2011 at 05:59:28AM -0400, Ayal Baron wrote:
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
We would still have an "SDM" which would prevent such a scenario from occuring. Management would have to choose the SDM. I don't see the problem.
I do not see how SDM helps us here, unless we give one of our SDMs control on our single VM DB. One Vdsm is asked to create VM-A. Another Vdsm is asked to create another VM, which happens to have the same name, but a totally different definition. Who decides which one is created?
In the standalone case, both VMs can be created (each VDSM is independent). Then, if one instance later tries to migrate a VM over there will be an error. This is not a problem for my usage scenarios.
On Mon, Sep 05, 2011 at 12:20:57PM +0300, Dan Kenigsberg wrote:
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
It is relevant. For example, we do not want two Vdsms creating different VMs with the same name or vmId. Your suggested scheme, where each Vdsm persists VM metadata on a storage domain that it sees, may lead to that.
I think this point underscores one of the differences between a RHEV-m managed environment and the stand-alone environment I am working on. In a RHEV-m data center, the manager must ensure that a VM can run on any host in the cluster. Therefore, it must prevent naming conflicts of the nature you are describing. In the standalone scenario, each host must be a completely independent entity with autonomy to create VMs as it sees fit. As a trade-off of that independence, there could be conflicts (or unavailable resources) during migration that can cause a fatal error. I would consider this to be perfectly normal in the standalone environment. An operator of a federated vdsm environment (two independent vdsm instances with migration capability) must be careful to choose unique vm names and ensure that shared storage is properly connected.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
The DB may be a mere directory. But it has to be ONE.
I think ONE per VDSM is okay (as long as we check for conflicts when importing VMs from another instance).
Thanks Dan and Ayal for your comments. I am catching up to the conversation after our long holiday weekend here in the US. Comments/questions below...
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
+1 to the caching of storage and networking info. This is the subject of another proposal I am working on. Right now, if I can remember the Storage Pool UUID that I was last using, I can recall all of the storage domains that were associated with it. Once I have these UUIDs, I can reconnect all of my storage again. I do want to make sure that once storage pools and the SPM go away, I will still have a means to recall my configured storage domains.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
Could you elaborate on what info for storage domains and networks must be localized? I would assume that for storage domains, there is the UUID and connection information. For networks, there is also a UUID and/or bridge name. Are these the only pieces of information?
If so, migration could work between two federated VDSM systems so long as a few rules were adhered to: - Shared storage is connected to both instances and accessed using the same UUID. - Equivalent networks share the same name and UUIDs.
For now, my standalone web interface could rely on these rules being followed as a matter of convention. In the future, logic could be added to the storage domain connection process whereby a previously initialized SD could tell the API user what its existing UUID is. Then, as new vdsm instances discover the shared storage, they will all access it via the same UUID.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
To be clear, you are suggesting that we could store this metadata on an existing DATA Storage Domain, right? In that case I agree. There is no need to create a new type of storage domain (META) whose exclusive purpose would be the storage of this type of information.
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
+1 to the caching of storage and networking info. This is the subject of another proposal I am working on. Right now, if I can remember the Storage Pool UUID that I was last using, I can recall all of the storage domains that were associated with it. Once I have these UUIDs, I can reconnect all of my storage again. I do want to make sure that once storage pools and the SPM go away, I will still have a means to recall my configured storage domains.
All connections info would be stored locally so no problem reaching domains, there is no conceptual difference between the 2.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
Could you elaborate on what info for storage domains and networks must be localized? I would assume that for storage domains, there is the UUID and connection information. For networks, there is also a UUID and/or bridge name. Are these the only pieces of information?
Wrt network info, I just realized it doesn't need to be stored locally. Whenever we store VM info on a domain, we can check what networks it requires and store the networking info on the domain (separately as many VMs will share the same networking info, no need to duplicate).
If so, migration could work between two federated VDSM systems so long as a few rules were adhered to:
- Shared storage is connected to both instances and accessed using the
same UUID.
- Equivalent networks share the same name and UUIDs.
Indeed.
For now, my standalone web interface could rely on these rules being followed as a matter of convention. In the future, logic could be added to the storage domain connection process whereby a previously initialized SD could tell the API user what its existing UUID is. Then, as new vdsm instances discover the shared storage, they will all access it via the same UUID.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
To be clear, you are suggesting that we could store this metadata on an existing DATA Storage Domain, right? In that case I agree. There is no need to create a new type of storage domain (META) whose exclusive purpose would be the storage of this type of information.
Correct. But it's not straightforward, it requires a new feature for existing domains (we've been discussing this for a long time).
-- Adam Litke agl@us.ibm.com IBM Linux Technology Center _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Wed, Sep 07, 2011 at 06:27:40AM -0400, Ayal Baron wrote:
To be clear, you are suggesting that we could store this metadata on an existing DATA Storage Domain, right? In that case I agree. There is no need to create a new type of storage domain (META) whose exclusive purpose would be the storage of this type of information.
Correct. But it's not straightforward, it requires a new feature for existing domains (we've been discussing this for a long time).
Thanks for the information Ayal. What is the best way for me to help with this? I know your team is busy with many other project requirements.
----- Original Message -----
On Wed, Sep 07, 2011 at 06:27:40AM -0400, Ayal Baron wrote:
To be clear, you are suggesting that we could store this metadata on an existing DATA Storage Domain, right? In that case I agree. There is no need to create a new type of storage domain (META) whose exclusive purpose would be the storage of this type of information.
Correct. But it's not straightforward, it requires a new feature for existing domains (we've been discussing this for a long time).
Thanks for the information Ayal. What is the best way for me to help with this? I know your team is busy with many other project requirements.
Saggi, can you braindump on this?
-- Adam Litke agl@us.ibm.com IBM Linux Technology Center _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Wed 07 Sep 2011 04:12:04 PM IDT, Ayal Baron wrote:
----- Original Message -----
On Wed, Sep 07, 2011 at 06:27:40AM -0400, Ayal Baron wrote:
To be clear, you are suggesting that we could store this metadata on an existing DATA Storage Domain, right? In that case I agree. There is no need to create a new type of storage domain (META) whose exclusive purpose would be the storage of this type of information.
Correct. But it's not straightforward, it requires a new feature for existing domains (we've been discussing this for a long time).
Thanks for the information Ayal. What is the best way for me to help with this? I know your team is busy with many other project requirements.
Saggi, can you braindump on this?
-- Adam Litkeagl@us.ibm.com IBM Linux Technology Center _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Your treading into very shaky grounds.
Just to make things clear there will never be another *new* domain class. Domain classes are a mistake and will be gradually removed. If some paradigm doesn't support the full features of a domain it shouldn't be one.
That being said I agree that there is a need for an API to push\pull "objects" from\to domains. These objects are blobs to VDSM and are only understood by management code. Further more I said many times that other objects like images and snapshots should also have a way to associate some metadata on them that is only understood by management.
This feature is so that VDSM will be more management agnostic. Anything that is not a direct responsibility of VDSM will be managed by objects pushed by management code. For interopability between management systems a standard will have to be created that VDSM will *not* enforce.
This will facilitate saving VM metadata on the domain. I continue to stress that VDSM will not be aware of it being a VM metadata.
As for connection management, I think that runtime management of connection information is the way forward. I am positively against VDSM reading the storage for additional connection or even understanding more metadata. I don't care if the mgmt pushes this as an object and uses it in it's own code.
As for how you can help, simplest thing would be to complete the missing functions in the current connection management code. public_validateStorageServerConnection public_getStorageConnectionsList
They are part of the current API and making them work will be a good start. The actual binding of connections to a domain is part of the new API and we are not sure how to go about implementing its support just yet. Also making iscsi.py. devicemapper.py multipath.py more feature complete and better separated is a good start for making sure we have the subsystems for advanced connection management features.
As for connection management, I think that runtime management of connection information is the way forward. I am positively against VDSM reading the storage for additional connection or even understanding more metadata. I don't care if the mgmt pushes this as an object and uses it in it's own code.
As for how you can help, simplest thing would be to complete the missing functions in the current connection management code. public_validateStorageServerConnection public_getStorageConnectionsList
They are part of the current API and making them work will be a good start. The actual binding of connections to a domain is part of the new API and we are not sure how to go about implementing its support just yet. Also making iscsi.py. devicemapper.py multipath.py more feature complete and better separated is a good start for making sure we have the subsystems for advanced connection management features.
Hi Sagi,
One of the things we want to improve in RHEVM 3.1 is the storage connection management. Today it is RHEVM responsibility to connect and disconnect the host from the storage domains explicitly, is that going to change as part of the new API? Is it going to be encapsulated in other verbs? Is there any documentation of that?
Thanks, Livnat
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Mon 12 Sep 2011 09:48:51 AM IDT, Livnat Peer wrote:
As for connection management, I think that runtime management of connection information is the way forward. I am positively against VDSM reading the storage for additional connection or even understanding more metadata. I don't care if the mgmt pushes this as an object and uses it in it's own code.
As for how you can help, simplest thing would be to complete the missing functions in the current connection management code. public_validateStorageServerConnection public_getStorageConnectionsList
They are part of the current API and making them work will be a good start. The actual binding of connections to a domain is part of the new API and we are not sure how to go about implementing its support just yet. Also making iscsi.py. devicemapper.py multipath.py more feature complete and better separated is a good start for making sure we have the subsystems for advanced connection management features.
Hi Sagi,
One of the things we want to improve in RHEVM 3.1 is the storage connection management. Today it is RHEVM responsibility to connect and disconnect the host from the storage domains explicitly, is that going to change as part of the new API? Is it going to be encapsulated in other verbs? Is there any documentation of that?
Thanks, Livnat
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
It is going to change. We are still discussing the best way to do it internally and there will be an email for peer review on the list once were done arguing.
vdsm-devel@lists.fedorahosted.org