Related to our disposable clients initiative in Taskotron, I'm working on ways how to
boot and manage those clients. In order to know how to boot them, I need to know how they
are created (e.g. cloud images require special magic with cloud-init). That's why I
spent some time in the past looking at different ways how to either create custom images
or re-use existing ones. The simplest approach turned out to be (in my view) a tool called
virt-builder. It's a part of libguestfs project and it's a super easy way to
create Fedora disk images - they have them already pre-created and stored on the server,
so it just downloads them and customize them a little bit per your requirements:
$ virt-builder fedora-21 --root-password password:fedora --install vim-enhanced --update
I decided to try to use it for creating the base image for our disposable clients. The
idea is that virt-builder is run once a day on a server and it generates a fresh new VM
disk image. We then use this disk image for all our disposable clients spawned in the
future, until a new disk image is generated the next day. Rinse and repeat.
The benefit of this approach (over e.g. official cloud images which might be updated once
per month or twice per year) is:
* the clients are always up-to-date
* it's very light on bandwidth - instead of downloading updates for every client on
every task run, it's done once per day, centrally
* it's much faster - the clients just boot and execute the task, they don't spend
time upgrading each boot
* it's less error-prone - there is a never-ending story of failures when you
communicate over network thousands times per day. We know it really well. By updating the
client just once, the chance for a network error is much lower, and we can easily repeat
it if it fails. Also, it's not a big deal if it completely fails some day, our tasks
continue to be executed without a problem, just with a day-old packages installed.
* can be integrated with fedmsg "updates pushed" notification to be almost
instantly in sync with the latest updates
* it helps with rescheduling tasks and reproducing issues - if we keep last X-number of VM
images, we can easily replay past tasks with exactly the same package set as it had during
Subsequently I realized that we will probably also want to have several different base
disk images, not just one. For example, certain GNOME checks might require most of GNOME
installed, which would mean downloading a lot of packages every time they are run. But
with virt-install, we can pre-create different base images for different purposes, and
then let the task define which image to use. So for GNOME checks, we would prepare a base
image with full GNOME installed.
To make all of this manageable, I created a wrapper around virt-builder and called it
taskotron-vmbuilder (ideas for a better name are welcome). The code is available here:
The concept is that you can define templates and then run vmbuilder which takes care of
the rest - it uses virt-builder to create the image, it makes sure that libtaskotron is
installed, it configures stuff like root password and ssh key. We can add any other
functionality that is needed. The simplest way to try this is this:
$ ./vmbuilder.py templates/f21-minimal.yaml
There are sample templates for minimal installation and Server and Workstation products.
For more details, look into READMEs, it's fairly well documented, I think. My idea is
that we will run this daily on the server, store the resulting images somewhere, and
update the pointer to the "current one".
I still haven't created project and repo in Phab, but if you approve this as a way
forward, I will.