Changes to 'ui_work'
by Kaleb KEITHLEY
New branch 'ui_work' available with the following commits:
commit 897a788f3383ddcdae3d704985f315754afc99a3
Author: Kaleb S. KEITHLEY <kkeithle(a)cloudfs-node01.kkeithle.redhat.com>
Date: Fri Apr 29 11:29:20 2011 -0400
checkpoint wip
12 years, 4 months
Changes to 'xattr-prefetch'
by Jeff Darcy
New branch 'xattr-prefetch' available with the following commits:
commit 3cd06b2b486fc59c3649c529e2da241feaca7165
Author: Jeff Darcy <jdarcy(a)redhat.com>
Date: Tue Apr 26 10:12:00 2011 -0400
Speed up directory listings by pre-fetching important xattrs.
12 years, 5 months
Thoughts about the "Dynamo" translator
by Jeff Darcy
Some of you might have heard me talk about the hypothetical-future
"Dynamo" translator. For those who haven't already seen my Summit
slides, the basic idea is to create a translator that (1) does
distribution in a more scalable way than the current DHT translator and
(2) combines distribution with replication and/or striping and/or
erasure codes. Some problems with the current DHT translator include:
* Global directory operations which limit scalability and performance.
* Placement of new files based on "stale" xattr data if the server set
has changed, until an global (and expensive!) rebalance operation is
triggered manually.
* Requirement that underlying stripe/replicate translators be statically
defined and not use overlapping sets of bare bricks, precluding many
otherwise-valid configurations (e.g. with "odd" numbers of bricks or
bricks with different capacities).
* Server-to-global inode number mapping which is based on the current
number of bricks and therefore does not guarantee stability when bricks
are added/removed (which might even result in clients attempting to
access the wrong file).
My goal is (eventually) to create a new translator which does
essentially the same things that DHT does, but does them in a way that
addresses the above issues. The basic design principles are mostly those
outlined in this document, which has become the basis for several
successful projects in the NoSQL space.
http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
I actually have a working 2.x translator (which I have at different
times called "swipe" or "scatter") which addresses most of the above
problems. It currently combines distribution and striping internally,
but I've since concluded that it would be better to combine distribution
and replication instead since the replication code is more in need of a
rewrite for performance reasons. Even better would be to have a coherent
mechanism that allows arbitrary combinations of striping and
replication, and that's where I run into a problem. The distribution
translator has to control the dynamic placement of data onto sets of
bricks which are then combined into stripe/replica groups, which creates
a coupling between these functions and is also at odds with the more
static way GlusterFS constructs the translator tree. I don't want to
create one big uber-translator which subsumes all three kinds of
functionality and might be harder to extend e.g. if/when we do erasure
codes or information dispersal instead. I also don't want to create a
private "plugin" API that programmers (i.e. those reading this email)
would have to learn in addition to the existing translator API.
On the drive in today, I think I came up with a way to satisfy this need
to keep things separate and uniform. The key is to take translators
which are written using the current translator API, but use them in a
very different way. Instead of configuring stripe/replicate translators
directly from the volfile, Dynamo would *dynamically* create
stripe/replicate translators corresponding to the sets of bricks or
layers of other stripe/replicate translators that it needs according to
its parameters and the current server topology. For example, if it needs
to do three-way striping across two-way replica sets, it might
dynamically create the following translators:
X = replicate(A,B)
Y = replicate(C,D)
Z = replicate(E,F)
S = stripe(X,Y,Z)
Other assignments are possible. The main point is that Dynamo would then
call S, which would call X/Y/Z, which would call lower-level translators
even though those are actually children of the original Dynamo
translator. This kind of "side call" is not something that GlusterFS
explicitly supports or does itself, but AFAICT there's nothing in the
way the translator "stack" or inode contexts etc. are managed that
prevent it from working. If necessary, we could work around some
differences by compiling regular translator code with redefined versions
of certain macros (especially STACK_WIND and STACK_UNWIND) to generate
"conformant" calls instead. That might even give us an opportunity to
perfom certain optimizations that the standard macros forego, so it
might be worthwhile even if it's not strictly necessary. The important
thing is that we should be able, using these techniques, to keep
striping/replication (and similar) functionality separate even under a
more dynamic distribution scheme.
12 years, 5 months
Re: options for implementing a cloudfsd and a UI (CLI and web UI) to drive it.
by Jeff Darcy
On 04/19/2011 03:57 PM, Kaleb S. KEITHLEY wrote:
> Here's a summary of the frameworks I've considered for implementing the
> client-server communication pathway, along with what I see as the pros
> and cons of each.
>
> My first choice would have been #3, but after further investigation I'm
> concerned that adding gsoap2 to RHEL7 could be problematic. I'm
> currently leaning toward option #2.
>
> Is it worth circulating this to a wider audience, e.g. cloudfs-devel?
>
>
> 1. pure web-based UI (CGI, python) and ssh, use any web browser as client
>
> Pro: lightweight, encrypted w/ https
> Con: lacks two-way SSL authentication, no CLI or hard to write one.
> License: none
I'm not sure if it's any harder to write a CLI with this approach than
with any of the others. We need to separate the "view" (command line or
web) from the "controller" anyway. That already implies some sort of
library interface to do things like fan requests out to multiple servers
that need to be involved in an action, and that library should be usable
either from CGIs or inline code (e.g. in the next approach). The lack
of two-way auth in this approach worries me more.
> 2. python bottle-based cloudfsd and curl-based client
>
> Pro: lightweight
> Con: more difficult to debug, doesn't seem to be suited for web UI,
> not directly anyway
> Notes: jdarcy says it will do SSL, i.e. https, it's not clear (to
> me) it will do two-way authentication, common parts could extracted
> and used by both a python-based CGI web UI and bottle-based cloudfsd
> accessed from a C+libcurl-based CLI
> License: MIT for bottle.py
I definitely know how to do (and have in fact done) two-way auth in a
bottle-based app. Integrating that with the existing cloudfsd code is
more a matter of standardizing key management than of actually changing
code.
> 3. gsoap2-based cloudfsd and client(s)
>
> Pro: high level APIs, has two way authentication
> Con: possible license issue, RPMs are available for F{14,15}, not
> RHEL (but it's in EPEL), some people complain that the generated
> stubs code is overly large
> Notes: web UI w/ CGI that invokes SOAP APIs (native python bindings
> to SOAP)
> License: GPL or gsoap public license (mozilla public license), wsdl
> and soap2cpp require a commercial license for commercial use. I
> infer that Red Hat would need a commercial license to generate use
> wsdl and soap2cpp to generate the stubs.
How high-level do we need this API to be? To put it another way, what
does gsoap (or xmlrpc etc.) allow us to do or express that's not as
easily done or expressed by passing REST requests and responses through
an SSL connection?
> 4. xmlrpc-based cloudfsd and client
>
> Pro: lighter than gsoap2.
> Con: not as high level as gsoap2, not clear whether it does SSL
> Notes: web UI w/ CGI that invokes xmlrpc APIs
> License: BSD-style, Expat, ABYSS, and others for some pieces.
>
> 5. xmlrpc3-based cloudfsd and client
>
> Pro: ?
> Con: Java-based and Apache Tomcat, long list of dependencies
> Notes: Web UI as with xmlrpc
> License: Apache
>
> 6. Aeolus Matahari
>
> Pro: part of Aeolus
> Con: don't know much about it
> Notes: but presumably there are people around who do
> License: presumed good
Not a big fan of Matahari in general, plus it's a more complex kind of
dependency than e.g. xmlrpc or bottle.
> 7. SNIA CDMI
>
> Pro: industry standard
> Con: written in Java, working draft is "...released so that
> developers can get started with integrating CDMI into their products..."
> License: BSD
As far as I know, CDMI is more geared toward operations on data objects
than on managing the details of the storage system serving those
objects. For most of the operations we care about - e.g.
adding/removing tenants, starting/stopping volumes, enabling/disabling
encryption - it seems like fitting them into framework would be either
cumbersome or impossible.
12 years, 5 months
Changes to 'uidmap'
by Jeff Darcy
New branch 'uidmap' available with the following commits:
commit 4c1d7c95e07f03b4800457fb1bfaf21a15b55d36
Author: Jeff Darcy <jdarcy(a)redhat.com>
Date: Tue Apr 19 12:36:03 2011 -0400
Initial version of UID-mapping translator.
12 years, 5 months
Branch 'cloudfsd' - scripts/volfilter.py
by Jeff Darcy
scripts/volfilter.py | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 50 insertions(+), 1 deletion(-)
New commits:
commit 928766a775474459a6bac9108506db3430264ea7
Author: Jeff Darcy <jdarcy(a)redhat.com>
Date: Tue Apr 19 11:46:27 2011 -0400
Changes got left out of previous cloudfsd-branch commit.
diff --git a/scripts/volfilter.py b/scripts/volfilter.py
index 98ab20e..1b61595 100755
--- a/scripts/volfilter.py
+++ b/scripts/volfilter.py
@@ -17,8 +17,52 @@
# You should have received a copy of the GNU Affero General Public License *
# along with CloudFS. If not, see <http://www.gnu.org/licenses/>.
+import copy
import string
import sys
+import types
+
+good_xlators = [
+ "cluster/afr",
+ "cluster/dht",
+ "cluster/distribute",
+ "cluster/replicate",
+ "cluster/stripe",
+ "debug/io-stats",
+ "features/access-control",
+ "features/locks",
+ "features/marker",
+ "performance/io-threads",
+ "protocol/client",
+ "protocol/server",
+ "storage/posix",
+]
+
+def copy_stack (old_xl,suffix,recursive=False):
+ if recursive:
+ new_name = old_xl.name + "-" + suffix
+ else:
+ new_name = suffix
+ new_xl = Translator(new_name)
+ new_xl.type = old_xl.type
+ # The results with normal assignment here are . . . amusing.
+ new_xl.opts = copy.deepcopy(old_xl.opts)
+ for sv in old_xl.subvols:
+ new_xl.subvols.append(copy_stack(sv,suffix,True))
+ # Patch up the path at the bottom.
+ if new_xl.type == "storage/posix":
+ new_xl.opts["directory"] += ("/" + suffix)
+ return new_xl
+
+def cleanup (parent, graph):
+ if parent.type in good_xlators:
+ sv = []
+ for child in parent.subvols:
+ sv.append(cleanup(child,graph))
+ parent.subvols = sv
+ else:
+ parent = cleanup(parent.subvols[0],graph)
+ return parent
class Translator:
def __init__ (self, name):
@@ -31,7 +75,12 @@ class Translator:
return "<Translator %s>" % self.name
def load (path):
- fp = file(path,"r")
+ # If it's a string, open it; otherwise, assume it's already a
+ # file-like object (most notably from urllib*).
+ if type(path) in types.StringTypes:
+ fp = file(path,"r")
+ else:
+ fp = path
all_xlators = {}
xlator = None
last_xlator = None
12 years, 5 months
Changes to 'cloudfsd'
by Jeff Darcy
New branch 'cloudfsd' available with the following commits:
commit 3be3fd214b18da3d3fa3e485fbfc13df1d74cecc
Author: Jeff Darcy <jdarcy(a)redhat.com>
Date: Fri Apr 15 10:55:32 2011 -0400
New code for daemon/port management.
12 years, 5 months