is there a depth limit for buckets?

Jim Meyering jim at meyering.net
Wed Mar 9 18:41:19 UTC 2011


Pete Zaitcev wrote:
> On Sat, 5 Mar 2011 18:25:33 -0600
> Steve Loranz <sloranz at redhat.com> wrote:
>
>> The instance of iwhd that I'm testing against seems to limit me to
>> one level of buckets.
>>
>> Storing a something at http://mywarehouse/unittests/object/metadata_key works
>> where trying to store it at
>> http://mywarehouse/username/unittests/object/metadata_key does not.
>>
>> Is this expected behavior?
>
> Hold on. First of all, a bucket is not a directory.
>
> I do not entirely understand why we inherited that. In S3, buckets
> are units of replication, reliability/durability/whatever acid, and
> performance. But in iwhd we replicate either the whole thing
> (for Mongo) or each data blob individually. Although, a bucket
> can have a replication policiy attached to it. Also, it may be useful
> for sysadmins that iwhd bucket maps 1:1 to the back-end's bucket,
> including S3. Your app knows where the data goes, it's a useful
> transparency property.
>
> Having said that, in S3, keys can have slashes in them. I do not see

Hi Pete,

>From the looks of the code (the way a URL is parsed),
I think iwhd's original design deliberately excludes the possibility
of any names (bucket, objects, attributes) including slashes.

If you want to add that as a wish-list item, that might be worthwhile
under the general "no arbitrary limitations" theme, but I don't see it
as a bug.  If you want to be sure, ask Jeff about his assumptions when
he wrote it.

> why you should not be able to store the same in iwhd, except that
> current filesystem backend has no clue about it and may blow up.
> We ought to fix this up for you pronto.
>
> In tabled, files are named as MD5 hashes of keys. This loses some
> essential transparency. If its database blows up, you'll never be
> able to find your data. In iwhd, you can, because keys are mapped
> to back-end filesystem properly. I like that, but it requires
> a fix, apparently.


More information about the iwhd-devel mailing list