Thanks Dan and Ayal for your comments. I am catching up to the conversation
after our long holiday weekend here in the US. Comments/questions below...
On Sun, Sep 04, 2011 at 03:16:08PM -0400, Ayal Baron wrote:
----- Original Message -----
> On Fri, Sep 02, 2011 at 03:44:58PM -0500, Adam Litke wrote:
> > If VDSM is to be functional in a stand-alone configuration, one of
> > the first
> > features that must be added is support for persistent domains. Since
> > there will
> > not be a management server around to "remember" domain
> > configurations, there
> > should be an option to have VDSM store it. Here is my idea for how
> > it could be
> > implemented. Your comments and advice are greatly appreciated.
> >
> >
> > Selecting a location to store VM metadata:
> > ------------------------------------------
> > The first design decision that needs to be made is where the VM
> > metadata should
> > be stored. Based on my investigation, there are two potential
> > locations: 1) In
> > a new directory under '/rhev/data-center', or 2) Inside the Master
> > Storage
> > Domain ('/rhev/data-center/<spUUID>/mastersd/master/vms/').
> >
> > Option #2 seems the most promising at first but I have a few
> > concerns. The
> > '/rhev/data-center/<spUUID>/mastersd/master/vms/' directory
already
> > exists
> > today. What is it currently being used for?
>
> RHEV-M uses this directory to hold a backup of its VM metadata. That's
> done via the updateVM verb. The metadata is stored there in OVF
> format.
>
> > Even though I have some VMs
> > created I do not see any information here. Also, if the concept of
> > Master
> > Storage Domain is going away, in the future how will we select a SD
> > in which to
> > store VM definitions?
> >
> > Option #1 may be an acceptable fallback if issues with #2 cannot be
> > resolved.
> > However, I would prefer that VM metadata be stored on managed
> > storage so that it
> > can be recovered if the host OS needs to be reinstalled.
>
> The deep reason why #2 cannot be resolved, is that VM metadata is
> essentially data-center-wide. Vdsm, however, is a single node manager,
> that may not have visibility to the complete network or storage
> domains.
Once we get rid of "master domain" and "SPM", each domain can have
its own are
for storing small files (e.g. VM definition file).
A VM definition should be stored on only one domain (true for both RHEVM use
case and any other use case) and if the VM has disks spanning multiple domains
then management/user needs to decide which domain will hold this info (e.g.
where the system disk is, or first domain which had a disk in it or whatever).
wrt Dan's comment, vdsm should have visibility to all the resources needed to
run a VM. This means that if vdsm is to run VM-X which has disks on storage
domain A and SD B then vdsm should be connected to both domains (same applies
to networks). vdsm could locally cache storage and networking info so that
when the VM references a storage or a network, we could automatically
connect/configure whatever we need.
+1 to the caching of storage and networking info. This is the subject of
another proposal I am working on. Right now, if I can remember the Storage Pool
UUID that I was last using, I can recall all of the storage domains that were
associated with it. Once I have these UUIDs, I can reconnect all of my storage
again. I do want to make sure that once storage pools and the SPM go away, I
will still have a means to recall my configured storage domains.
The "data-center-wide" bit is totally irrelevant as the
storage is shared so
as long as vdsm can access the domain, it's fine.
Only issue that needs to be resolved is how different nodes get the
networking/storage connections info localized in the first place (obviously we
can save whatever we see when a VM is run, but other vdsms are expected to
know this info regardless. I believe this indeed would be the job of another
layer, because vdsm as you say is single node and *this* operation is
migration cluster wide.
Could you elaborate on what info for storage domains and networks must be
localized? I would assume that for storage domains, there is the UUID and
connection information. For networks, there is also a UUID and/or bridge name.
Are these the only pieces of information?
If so, migration could work between two federated VDSM systems so long as a few
rules were adhered to:
- Shared storage is connected to both instances and accessed using the same
UUID.
- Equivalent networks share the same name and UUIDs.
For now, my standalone web interface could rely on these rules being followed as
a matter of convention. In the future, logic could be added to the storage
domain connection process whereby a previously initialized SD could tell the API
user what its existing UUID is. Then, as new vdsm instances discover the shared
storage, they will all access it via the same UUID.
>
> This is also why I am reluctant to add VM database into Vdsm - I think
> it belongs to a higher level. Handling the removal of a defined VM on
> migration success seems very hard to do right - imagine destination
> crash just in the worst time. We need a distributed database for that.
>
> Single node use case is very important, though. I would love to see a
> Vdsm sub-package that maintains a database of VM domains, as well of
> all
> other inter-node information.
No reason for this to be a separate DB and not just a local storage domain.
To be clear, you are suggesting that we could store this metadata on an existing
DATA Storage Domain, right? In that case I agree. There is no need to create a
new type of storage domain (META) whose exclusive purpose would be the storage
of this type of information.
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center