On Fri, 2018-11-23 at 13:43 +0000, Richard W.M. Jones wrote:
On Fri, Nov 23, 2018 at 12:48:30PM +0000, Peter Robinson wrote:
> On Fri, Nov 23, 2018 at 12:43 PM Richard W.M. Jones <
> rjones(a)redhat.com> wrote:
> > It looks like the raw format xz-compressed cloud images that we
> > ship
> > use a very large block size, possibly 192M. This is not ideal
> > and it
> > would be better to use a smaller block size such as 16M so that
> > they
> > can be consumed without having to be uncompressed by nbdkit, or
> > even
> > be used remotely without even downloading them.
> >
> > (Long story here:
> >
https://rwmj.wordpress.com/2018/11/23/nbdkit-xz-curl/ )
> >
> > I recompressed the Fedora 29 cloud image using a 16M block size
> > and
> > there is about 4% overhead to doing this:
> >
> > before 194278292
> > after 202874868
> >
> > At the moment I'm not clear what actual component does the xz
> > compression step, so I don't even know where I could file a bug
> > or who
> > I could discuss this with, nor what the current code looks like.
> > Apparently it's no longer done using appliance-tools.
>
> The cloud images haven't used appliance-tools for years. It uses
> imagefactory and some bits in koji.
>
> Looking at the output of the logs it just runs:
> $ /usr/bin/xz -z9T2
> /var/tmp/koji/tasks/2418/31062418/Fedora-Cloud-Base-Rawhide-
> 20181123.n.0.aarch64.raw
>
> Example task:
>
https://koji.fedoraproject.org/koji/taskinfo?taskID=31062418
>
> So at a guess it should be a straight forward patch to koji.
Thanks Peter, I found the source now:
https://pagure.io/koji/blob/master/f/builder/kojid#_3887
More generally, what are the goals for these cloud images? Example
questions for the mailing list:
Must they be absolutely as small as possible? (I assume small size
is
a goal to some extent because of the cost of bandwidth and mirroring
space).
Is it important to let people download them and run them without
uncompressing them? For me, unxz is pretty slow as well as the
obvious problem with consuming disk space, so I can get started on a
cloud image faster if it can be uncompressed on the fly (faster still
if I didn't have to download it up front, but that has other issues
like causing load on the mirror sites).
$ time unxz Fedora-Cloud-Base-29-1.2.x86_64.raw.xz
real 0m23.760s
user 0m21.564s
sys 0m0.729s
What do people do with the cloud images? Do you download them and
put
them in local a Glance store? Do you ignore the published cloud
images and instead get Fedora on clouds through some other method
like
AMIs published by 3rd parties?
Primarily they are there for use in uploading to
cloud providers. When
we set things up our goal was solely to make them as small as possible,
our only use case was uploading to EC2 and we delivered them as part of
the release solely to make sure they were available and people could
compare what was in EC2 to what is available for verification
processes. The only other use case we considered was people downloading
and putting them in their own cloud environment. It sounds like you
have some other use cases and should work with release engineering to
look at accomodating them. We probably should have some documentation
on each deliverable and the use cases they are supposed to support.
Dennis