As we get closer to putting disposable clients into production, we
need
a way to have updated images for those clients. I don't think this is
news to anyone since the topic has come up several times before but now
there's a bit more urgency :)
In my mind, we have the following requirements:
- Produces qcow2 images that work with testcloud
- can be run in an automated way
- allows adding/changing/customizing packages contained in image
- allows arbitrary repos to be specified
and the following "nice to have" things:
- can build branched and rawhide images
- builds images from scratch using only things provided by releng
- written in python
- builds more than qcow2 for some future-proofing
qemu-img can convert between many image formats, so native support in that tool is not
that important, I think.
- can run well in a VM
Is there anything that I missed?
The image should be compatible with guestfish, so that we can e.g. copy in some files
without rebuilding the image from scratch. Might be useful for e.g. additional ssh keys
(we have cloud-init for that at the moment, but if we had some troubles with it or we
needed something it doesn't support, this would be an alternative way). I'm not
fully sure what the requirements are, but I think guestfish can work with almost anything,
including LVM, so unless the tool creates some crazy partition layout, it should work with
everything.
As far as I know, we're looking at two options right now:
taskotron-vmbuilder and imagefactory. I've put together a list of
the pros and cons that I know of for both tools. Thoughts on which
direction to take would be appreciated.
Tim
taskotron-vmbuilder [1] is a PoC system kparal built around
virt-builder [2]. Images are specified in a yaml file and instead of
building those images from scratch "It takes cleanly prepared, digitally
signed OS templates and customizes them".
[1]
https://bitbucket.org/fedoraqa/taskotron-vmbuilder
[2]
http://libguestfs.org/virt-builder.1.html
pros:
- already does almost everything we need
To be fair, there have been some issues regarding SELinux, and I'm not sure they are
sorted yet. The SELinux contexts of files inside the image were not set properly and one
more reboot with autorelabel was needed. Might be fixed now, or not, I haven't tried
for a long time. With anaconda, we're not likely to hit these kind of issues
(we'll hit different ones).
- fits all requirements
- builds quickly
- well supported
cons:
- requires blobs which are out of our control
* yes, I know who does the work behind virt-builder. My concern
isn't with him, it's the general concept that I don't like. This
also gets into the fact that we would have pretty much no control
over timing of release for the base images.
All the tools required to create that "blob" - or image template (as they call
it) - are open source and in Fedora, from what I see, so we can host our own. virt-builder
man page says:
"For serious virt-builder use, you may want to create your own repository of
templates."
This is how to create the template:
http://libguestfs.org/virt-builder.1.html#create-the-templates
For new stable Fedora releases, we can a single install manually, and use virt-sysprep on
the image to have it ready. For Rawhide and maybe even for Branched, we might want to
prepare a fresh new template more often, i.e. automate that. This is how libguestfs
project does it:
https://github.com/libguestfs/libguestfs/blob/master/builder/website/fedo...
So you might see this as a combination of imagefactory and virt-builder-style process. The
image is installed clean using anaconda once in a time (but very rarely), and most of the
time just the prepared template is adjusted (updated with new packages), because it's
much faster.
I'm not saying this is better or worse than alternatives, I just don't think this
"blob" argument is quite right - we'd probably create and host our own
templates, not rely on upstream one.
- limited support for rawhide and branched releases
There's limited (or no) support for it in the upstream repo, that's correct.
But if we host our own repo, according to the documentation and source code, it seems that
as long as anaconda can install it, it should be possible to create an image for it. Which
sounds as the same situation as with imagefactory. (Of course with the additional
requirement that virt-* tools have to work in Rawhide/Branched).
- limited support for non-server spins
I'm not really sure what you mean, we can install any package set we want, so the only
difference would be in the filesystem layout? The upstream templates seems to have only
@core installed, in our own images we could adjust even that.
- output images are large in size
This is interesting. Theoretically I see no reason why official Cloud images should be
smaller than the same package set installed using virt-builder. I guess they are simply
more stripped down, and the filesystem much smaller? It could use some investigation.
It's also a question how imagefactory-created images will look like (once we use our
custom kickstarts).
By the way, since we seem to agree that we'll need several package set templates for
each release (minimal, server, workstation), we're going to distribute pretty big disk
images anyway. (Which concerns me a bit in itself).
- virt-builder is not written in python
Yeah, there are some parts of it in OCaml. Scary. I wouldn't want to patch that :)
imagefactory [3] is a system for building os images and potentially
shipping those images to various cloud systems. Images are specified
with a kickstart file and an xml template descriptor. Imagefactory
builds images from scratch, essentially using the kickstart to run an
install inside a VM and processing that install into the desired image
type.
[3]
http://imgfac.org/
pros:
- used by releng to create Fedora cloud images
Collaborating on the tool with another team would be a big win from my POV.
- builds images from packages: no blobs that we don't have
control over
- already has a mostly-complete RESTful api that can list images and
trigger new builds
- can support almost all spins; anything that can be represented in a
kickstart
- written in python
cons:
- not as fast as virt-builder
I haven't tried imagefactory, but we all know how long anaconda installation takes. I
don't think it's a problem for our production environment, we don't care about
10 minutes difference. But if we consider building the same image on task-developer's
machine, the speed gets more important.
- somewhat more complex than virt-builder
That's true, but if we start prepare our own virt-builder templates, I think that
quickly reaches parity in complexity.
- when something goes wrong, debugging can be difficult due to how
the
tool works
Do they have something like real-time monitoring of anaconda logs, do you know? Because
otherwise I guess it's quite hard to learn what went wrong.
- we may be somewhat on our own to fix issues if releng is not
hitting
similar problems
- may not run well in a VM (would need nested virt)
This is the same as in virt-builder, it also needs virt support. Originally I thought it
doesn't, but it does. It can still be used without hw virt support (unlike anaconda,
that would just be impossible performance-wise), but it's much much much slower and I
don't think we would want to go that route (building an image 30 minutes instead of 3
minutes).