----- Original Message -----
Ayal, thank you for your explanation. This really helps to
understand
what is
planned for the storage architecture revamping.
A few questions: Can the role of SDM transition between multiple hosts
in the
same manner as SPM does today? Can a host have SDM responsibilities
for
multiple storage domains?
Yes on both accounts.
Based on your explanation of the planned architecture, I'd throw my
weight
behind option 3 (the simpler, smarter API). I realize I might not
fully
appreciate the complexity involved when taking that approach.
That was my initial thought too, but the more I think about it the less I like it.
Having management call create and then copy has many advantages:
1. No need for hosts to communicate with each other - in option 3, communication would be
done through the storage as is done today for lvextend flow so there is some latency
involved or else we would need to introduce a network dependency which we do not require
today.
2. one host or one storage disappearing would not prevent rollback. This to me is the
crux of it. As long as manager is around then it can persist the transaction data and
then at any given point in the future can clean things up as resources become available
again after error. Managing a transaction across 2 disparate storage targets is
impossible (there is just no atomicity) so you will always find cases where you're
left with garbage on your domain with no way of knowing you need to clean it up.
Disadvantages
2 ops instead of one - API a bit less intuitive, but this is a clustered op so I have no
problem for manager to deal with this actually, as to me, this is exactly the boundary
between vdsm and an external manager. vdsm does not manage the cluster, only the node.
On Thu, Sep 01, 2011 at 02:11:04AM -0400, Ayal Baron wrote:
>
>
> ----- Original Message -----
> > On 08/18/2011 01:31 AM, Saggi Mizrahi wrote:
> > > There is currently no way to list available unconnected pools.
> > > Pools
> > > are managed by the management. In the future we are going to
> > > remove
> > > the
> > > pool idiom from VDSMs API and only let the management side deal
> > > with
> > > it. Use the managements REST API to get the known pool list. It
> > > will
> > > also return pools that the current VDSM might not even know
> > > about
> > > yet.
> >
> > I suppose I should clarify my project a bit. I am working on
> > adapting
> > vdsm for use as a standalone node-level virtualization API. In my
> > environment, there is no upper management layer. I would like
> > users to
> > be able to browse a list of pools and 'activate' one in order to
> > gain
> > access to its associated storage domains.
> >
> > In absence of an enumeration mechanism, maybe I will create
> > hard-coded
> > UUIDs that represent a storage pool for each type of storage
> > (LOCAL_FS,
> > NET_FS, ISCSI, etc).
> >
> > Could you provide a little bit more detail on your plans with
> > respect
> > to
> > removing the Storage Pool concept from VDSM? How will that affect
> > the
> > current API?
>
> I see this got no real answer so I'll add some info here.
> Today's "pool" creates an arbitrary dependency between different
> storage targets (storage domains). Following is an explanation of
> the dependency:
> Currently, only a single node in the cluster (the "SPM") is allowed
> to make changes to the structure of all domains in the pool (create
> new LVs, add LUN to VG, etc). In order to achieve this, it is
> required to store MD about the domains in the pool.
> This metadata, referred to as the pool MD is stored on one of the
> domains and that domain is defined as the "master" domain. From this
> point on, in order to make changes to one domain, the master domain
> has to be available hence now the availability of all domains
> depends on this one domain. If it is unavailable we need to start
> taking corrective action (directed by mgmt) and this causes all
> kinds of ugly error flows.
>
> Getting rid of the pool would mean that instead of having a single
> manager (SPM) for all domains, each domain would be managed
> separately by an SDM (Storage Domain Manager).
>
> This not only eliminates the dependency but also allows for
> additional flexibility, because now instead of cross-domain
> operations being limited to the domains in the pool, operations can
> be performed between any 2 domains a host can access.
>
> The only operation that actually requires 2 domains is when copying
> an image between the domains (let's ignore VMs that have disks on
> multiple domains for now).
> Today, this operation is run as one atomic operation on the manager.
> Obviously, once there are 2 (possibly) disparate managers for the 2
> domains, this can no longer be the case.
> So to achieve this there is a need for a change in architecture.
> To understand this let's examine the copy image flow:
> In general we can split the flow into 2 phases:
> 1. create target placeholders (e.g. create LVs to store data)
> 2. copy data from source to destination
>
> For example, we have 2 nodes in our cluster.
> Each node has a local storage domain and in addition there is a
> shared iSCSI SD.
> Now user wants to copy an image from local storage to the shared
> storage.
> Obviously only one host has access to the local storage, but that
> host is not necessarily the manager (SDM) of the shared domain.
>
> So to copy the image we need the manager to create the placeholders
> and then the host with access to both domains to do the actual data
> copy.
> The architectural change required here is to actually split the
> operation in 2.
> This can be achieved by:
> 1. user (upper mgmt layer) calling operation 1 on SDM and then calls
> operation 2 on host
> - Requires upper management to understand internals of the
> operation, but enables a single point of managing the transaction.
> 2. user calls "copy" op on SDM. SDM creates the place holders and
> then communicates with the relevant host and tells it to copy data -
> user needs to communicate with a single node (simpler API), but
> requires both hosts to be up for the entire duration of the
> operation (host with local storage obviously, but now also SDM) and
> requires the SDM to be able to access all storage domains to/from
> which we want to copy images.
> 3. user calls "copy" op on the node with the local storage. The node
> requests the SDM to create placeholders (this is a quick operation)
> and once done proceeds to copy the data (very long operation).
>
> I think it is clear that option number 2 is worst and can be struck
> out.
> The decision between options 1 and 3 however is not that clear cut.
> Currently we are leaning towards a simpler API (option 3) but this
> is not finalized yet.
>
> The problem is error handling flows. Basically if copy fails after
> the placeholders have been created, we need to rollback. The
> transaction information needs to be persistently stored somewhere
> (e.g. on the local SD). This means however, that if the problem is
> that the host from which the op is run can no longer access the
> shared storage we will be left with garbage on the shared domain
> without knowing that we need to collect it.
> If the transaction is managed by the central manager then either
> domain disappearing (e.g. the source domain burns down during the
> op, or the target domain going down for a few minutes) can be
> handled later (once destination domain is available again, instruct
> the manager to remove the placeholders).
>
> Hope this helps.
>
> >
> > Thank you for the information.
> >
> > --
> > Adam Litke
> > IBM Linux Technology Center
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel(a)lists.fedorahosted.org
> >
https://fedorahosted.org/mailman/listinfo/vdsm-devel
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/vdsm-devel
--
Adam Litke <agl(a)us.ibm.com>
IBM Linux Technology Center
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel