Hey, folks. So, as per
https://fedorahosted.org/fedora-qa/ticket/307
, I
think we need to add a process for validating the multi-live and
multi-install images that we've been building for the last few
releases.
It is ironic that before F17 release I have talked about this to cwickert (IIRC) and he
assured me no QA help is needed and that nothing can go wrong. When they build the image,
they just ensure everything boots and that's it. There's just a small syslinux
glue, nothing else is changed. Well, it didn't go according to the plan, apparently.
I've put together a draft matrix template for this validation
here:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_va...
It's similar to the other validation matrices, and the changes are
mostly obvious. The key bit is the layout of the matrix and the test
cases themselves. I haven't actually written any of the specific test
cases yet, but I tried to think of the best way to organize things.
We
obviously need to do the basic sanity tests on each image, then we
need
to test that each image boots and the menu that lets you select which
sub-image to boot works properly, and also that each sub-image boots
and
installs properly. I think that set of tests ought to pretty much
cover
the intended functionality of the compendium images. When actually
writing the test cases, we might be able to come up with ways to take
advantage of some of the existing install/base/desktop validation
test
cases, but I haven't quite thought that far ahead yet.
I tried to organize the planned test cases as efficiently as
possible;
this is the best layout I could come up with, but someone might be
able
to suggest improvements. In case it's not obvious, here's roughly
what
each test case does:
QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums
-
these test cases already exist and can apply without modification to
the
compendium images, they simply check that they're within required
size
limits (dual-layer DVD size for these images) and the published
checksums match the images.
QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will
simply check that the compendium images themselves boot
I don't understand what this means.
To clarify, multidvd structure is this:
-> default DVD boot (auto-selects architecture)
-> select arch manually -> i386
-> x86_64
-> basic video -> both 2 options
-> memory test
-> local drive boot
So you probably mean the "default DVD boot" here, right? But which architecture?
The result might differ on i386 and on x86_64 machine.
But multilive structure is this:
-> default Live GNOME (auto-selects architecture)
-> default Live KDE (auto-selects architecture)
-> default Live LXDE (auto-selects architecture)
-> default Live SoaS (auto-selects architecture)
-> default Live XFCE (auto-selects architecture)
-> select arch manually -> i386 -> Live GNOME
-> Live KDE
-> Live LXDE
-> Live SoaS
-> Live XFCE
-> x86_64 -> Live GNOME
-> Live KDE
-> Live LXDE
-> Live SoaS
-> Live XFCE
-> basic video -> all 10 options here
-> memory test
-> local drive boot
What does the test refer to for multilive?
when written
to
a physical dual-layer DVD and booted as normal.
How important is here to really burn it? Is ISO VM booting regarded as a reliable test or
not? (Note: We don't burn classic Fedora images when testing either.)
We _may_ also want to
test booting them when written to USB using one specific method.
I don't think so. This is just to be pressed to DVDs. If Red Hat ever starts to give
away USB sticks, we can reconsider. Until then, it's redundant work.
I
don't
think we need to test any other way of booting the images, as these
images are specifically intended to be written to physical media and
given out at events; we don't really intend people to download them
and
write/boot them via all possible methods themselves, though they can
do
that if they really want to. So we don't really need to cover VM
booting
of the image, writing to USB via all other possible methods and so
on.
QA:Testcase_multilive_menu and QA:Testcase_multidvd_menu - these will
test that the top-level menus on each image, for picking which
sub-image
to boot, function correctly. I'm not sure at present if it'll be
possible to just instead write a generic QA:Testcase_multi_menu test
case which would apply to either image; if so, we can do just
QA:Testcase_multi_menu and QA:Testcase_multi_boot test cases and have
one result column for each image (as in the Image sanity table)
instead
of four test cases and a single result column.
QA:Testcase_multi_sub_boot and QA:Testcase_multi_sub_install - these
will test that each sub-image on each compendium image boots and
installs correctly. It would probably be unusual for _just one_
sub-image to fail, so we could probably get away with just testing a
single sub-image on each medium, but everything that can go wrong
will
go wrong, so we might as well cover everything if we have
time/resources. We may be able to re-use an existing test case
instead
of writing new ones here, I'll have to check the exact wording of
some
of our existing cases.
Thoughts, comments, objections, suggestions? Can everyone see where
this
is going, or am I zooming off up the wrong tree? Can you think of a
better approach? Thanks!
I have to say I'm somewhat confused from the current matrices structure. But maybe
when the test cases are written it'll clearer.
My idea personally was:
MultiLive matrix | GNOME | KDE | LXDE | SoaS | XFCE
------------------------------------------------------------------
multilive_auto_boot (i386) |
multilive_auto_boot (x86_64) |
multilive_manual_boot (i386) |
multilive_manual_boot (x86_64) |
multilive_install (anyarch) |
"auto_boot" is when you select the default option and correct architecture is
chosen for you. To test this properly, you need both i386 and x86_64 machines (or VMs, if
accepted).
"manual_boot" is when you select the architecture manually from the submenu. A
single x86_64 machine is sufficient to check both cases.
MultiDVD matrix | result
---------------------------------------
multidvd_auto_boot (i386) |
multidvd_auto_boot (x86_64) |
multidvd_manual_boot (i386) |
multidvd_manual_boot (x86_64) |
multidvd_install (anyarch) |
I am wondering whether we need installation tests at all. Theoretically the ISOs are
unchanged so there should be no difference at all. If they start to boot, it should be
fine. But according to Murphy's Law...