Thanks Dan and Ayal for your comments. I am catching up to the conversation after our long holiday weekend here in the US. Comments/questions below...
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
If VDSM is to be functional in a stand-alone configuration, one of the first features that must be added is support for persistent domains. Since there will not be a management server around to "remember" domain configurations, there should be an option to have VDSM store it. Here is my idea for how it could be implemented. Your comments and advice are greatly appreciated.
Selecting a location to store VM metadata:
The first design decision that needs to be made is where the VM metadata should be stored. Based on my investigation, there are two potential locations: 1) In a new directory under '/rhev/data-center', or 2) Inside the Master Storage Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
Option #2 seems the most promising at first but I have a few concerns. The '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory already exists today. What is it currently being used for?
RHEV-M uses this directory to hold a backup of its VM metadata. That's done via the updateVM verb. The metadata is stored there in OVF format.
Even though I have some VMs created I do not see any information here. Also, if the concept of Master Storage Domain is going away, in the future how will we select a SD in which to store VM definitions?
Option #1 may be an acceptable fallback if issues with #2 cannot be resolved. However, I would prefer that VM metadata be stored on managed storage so that it can be recovered if the host OS needs to be reinstalled.
The deep reason why #2 cannot be resolved, is that VM metadata is essentially data-center-wide. Vdsm, however, is a single node manager, that may not have visibility to the complete network or storage domains.
Once we get rid of "master domain" and "SPM", each domain can have its own are for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use case and any other use case) and if the VM has disks spanning multiple domains then management/user needs to decide which domain will hold this info (e.g. where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to run a VM. This means that if vdsm is to run VM-X which has disks on storage domain A and SD B then vdsm should be connected to both domains (same applies to networks). vdsm could locally cache storage and networking info so that when the VM references a storage or a network, we could automatically connect/configure whatever we need.
+1 to the caching of storage and networking info. This is the subject of another proposal I am working on. Right now, if I can remember the Storage Pool UUID that I was last using, I can recall all of the storage domains that were associated with it. Once I have these UUIDs, I can reconnect all of my storage again. I do want to make sure that once storage pools and the SPM go away, I will still have a means to recall my configured storage domains.
The "data-center-wide" bit is totally irrelevant as the storage is shared so as long as vdsm can access the domain, it's fine.
Only issue that needs to be resolved is how different nodes get the networking/storage connections info localized in the first place (obviously we can save whatever we see when a VM is run, but other vdsms are expected to know this info regardless. I believe this indeed would be the job of another layer, because vdsm as you say is single node and *this* operation is migration cluster wide.
Could you elaborate on what info for storage domains and networks must be localized? I would assume that for storage domains, there is the UUID and connection information. For networks, there is also a UUID and/or bridge name. Are these the only pieces of information?
If so, migration could work between two federated VDSM systems so long as a few rules were adhered to: - Shared storage is connected to both instances and accessed using the same UUID. - Equivalent networks share the same name and UUIDs.
For now, my standalone web interface could rely on these rules being followed as a matter of convention. In the future, logic could be added to the storage domain connection process whereby a previously initialized SD could tell the API user what its existing UUID is. Then, as new vdsm instances discover the shared storage, they will all access it via the same UUID.
This is also why I am reluctant to add VM database into Vdsm - I think it belongs to a higher level. Handling the removal of a defined VM on migration success seems very hard to do right - imagine destination crash just in the worst time. We need a distributed database for that.
Single node use case is very important, though. I would love to see a Vdsm sub-package that maintains a database of VM domains, as well of all other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
To be clear, you are suggesting that we could store this metadata on an existing DATA Storage Domain, right? In that case I agree. There is no need to create a new type of storage domain (META) whose exclusive purpose would be the storage of this type of information.