On 09/22/2011 07:25 PM, Matt Wagner wrote:
Hi all,
There's been some discussion about the fact that "aeolus-image list
--images" is extremely slow. It's fine with a handful of images, but the
time required grows exponentially, so that by perhaps 25 images, the
command takes minutes or hours to run.
The problem is that we tried to implement associations like they existed
natively -- an Image has many Builds. But since iwhd stores this
information backwards from how we want it, we need to load ALL builds
and then match the one with our image_id in it.
Where this gets outrageously inefficient is for things like the
"aeolus-image list --images" use case, where we have to walk all
associations many times. (Although in theory we're listing "images," the
reality is that we need several other data types to get all the
requisite data. Otherwise we're just listing UUIDs.) It ends up that,
for each image, we load all builds, and, for each of those builds, we
load all target images, and for each of those, we load all provider
images. This is fine for small sets, but the number of requests grows
exponentially and becomes absurdly impractical very quickly.
The "real" fix that we've kicked around is to find a new way to store
this information. And we should do that at some point. But it's clear
that this isn't going to happen soon, and I think we need to take some
action now.
Here's what I propose: if we know we're going to have to traverse any
associations, load all the objects of that type into memory for the
duration of the request, and do the matching in memory.
So for "aeolus-image list --images", we would instead load all images,
builds, target images, and provider images once, and then do all of the
mapping in memory. This can save a few orders of magnitude of API calls,
and, because it's only for the duration of the request, doesn't leave us
caching stale data.
I haven't fully thought through how to implement this yet, as we don't
want to load these associations if they're not needed at all. But as
soon as we know we're going to want an association, we should load all
of them into memory.
This isn't a perfect plan by any stretch, but I think it makes things
substantially less-bad, and should serve at least as a stop-gap.
What do you all think of this approach?
Best,
Matt
Hi,
this is just an idea which I didn't investigated further, but doesn't
iwhd support queries now? I mean I thought it's possible to query search
for bucket objects by attributes. From warehouse_api_final.md:
The syntax to issue a query is as follows.
$ curl -d '($color == "green") && ($flavor ==
"lemon")' \
http://fserver-1:9090/my_bucket/_query
<objects>
<object>
<bucket>my_bucket</bucket>
<key>file1</key>
</object>
</objects>
I remember this wasn't implemented when Tomas was working on warehouse
ib, but I think I noticed something like this in imagefactory.log so
maybe it's supported now. Not sure about this but I think it's worth
checking this because then number of queries reduces to ~3 per once image.
Jan