On Fri, 2014-11-07 at 13:02 -0500, Kamil Paral wrote:
I have a proposal regarding installation matrix [1] related to
environments. Adam did a truly stellar job in his latest results table
restructuring, he eliminated a lot of duplicated testing and
simplified the tables a lot (for example, there's no point in testing
additional HTTP repos in anaconda on all three major architectures,
because that functionality it architecture independent). We don't have
limitless resources and must keep focused on areas where it's likely
for a bug to occur, instead of testing everything. Here's a proposal
of some last touches to installation tables:
1. Testcase_Anaconda_User_Interface_Basic_Video_Driver was changed to
a generic x86 test. In this case, it is problematic, because BIOS and
UEFI fallback video driver are completely different code paths. I have
a personal experience from that past when one of those worked and the
other didn't. We should distinguish BIOS and UEFI in this case. To fix
this, I see two viable options:
a) move the test case to Miscellaneous section, so that the table in
User interface section doesn't get more complicated
b) keep the test case in User interface section, but create a
separate
table for it (right under the first table). This new table would have
x86 BIOS and x86 UEFI columns. Again, the purpose is to keep the
original table simple (avoid adding columns which are not required for
anything else).
The tests in that table are really covering somewhat different things,
now I look at them. Graphical, and Text actually exercise two of
anaconda's UIs (we seem to be missing a test for cmdline, unless one of
the other test cases hits it in passing). The others are different ways
of accessing those same UIs: the VNC tests are different ways to get at
graphical, and serial_console is another way to get at text.
It makes sense for Graphical and Text to not be split between i686,
x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to
have probably x86 BIOS / x86 UEFI split, indeed. We should consider the
other tests in deciding the right thing to do - what's the appropriate
split for VNC and serial console?
2. Testcase_Install_to_Previous_KVM and
Testcase_Install_to_Current_KVM differentiate BIOS and UEFI. I always
forget whether this is about the host or the guest (or both, or which
combination). But I think host differentiation doesn't make any
difference - libvirt should work the same, I assume. It makes sense to
differentiate the guests. Unfortunately I believe the UEFI mode for
libvirt/kvm [2] is not available under an open source license,
therefore it's not packaged and not trivial to use. Which means not
that many people probably use it. I wonder whether we really need to
keep this separated. I would gladly merge this together to a single
x86 column and split up once OVMF is available under a better license.
If we keep it separate, we should provide better instructions for the
UEFI part.
Well, the licensing issue is a bit subtle. OVMF itself is freely
licensed (BSD or MIT IIRC). The bit that has a license issue is the FAT
driver used by EDK II, on which OVMF is built. It has this extra term in
its license:
Additional terms: In addition to the forgoing, redistribution and use
of the code is conditioned upon the FAT 32 File System Driver and all
derivative works thereof being used for and designed only to read
and/or write to a file system that is directly managed by Intel's
Extensible Firmware Initiative (EFI) Specification v. 1.0 and later
and/or the Unified Extensible Firmware Interface (UEFI) Forum's UEFI
Specifications v.2.0 and later (together the "UEFI Specifications");
only as necessary to emulate an implementation of the UEFI
Specifications;
and to create firmware, applications, utilities and/or drivers.
which obviously makes it non-free. I believe the *reason* is something
to do with Microsoft patent(s) believed to be related to FAT, and I
don't think it's prudent to say anything more specific than that.
There therefore isn't a legal issue with redistributing OVMF builds, and
there is a commonly-used repository which contains them, updated daily:
https://www.kraxel.org/repos/
they just can't go into Fedora, because of Fedora's policies about
non-freely-licensed software. It's pretty easy to set up a UEFI KVM
using the packages from that repo, I have one here and just spent all
morning using it to test UEFI multi-boot.
We certainly *can* block Fedora releases on things that have to do with
software that's not part of Fedora's repositories, if we want to - viz
the Windows or OS X boot criteria, for e.g. It'd be interesting to see,
if a blocker was proposed, if we actually want to block on UEFI KVM
guest operation, but I think at least it's worth having it in the
matrix.
PS: I know nothing about Xen, which is also in the table, so I have
no
idea whether it's a similar story or not.
We have a commitment from Oracle to provide the testing for that
criterion - basically we agreed to have the criterion so long as they
make sure the testing gets done. Konrad Wilk has been doing that testing
ever since we added the criterion, and he usually does it promptly and
thoroughly, so we haven't really had any trouble with it.
3. Testcase_Anaconda_save_traceback_to_bugzilla,
Testcase_Anaconda_updates.img_via_URL,
Testcase_Anaconda_user_creation,
Testcase_Anaconda_updates.img_via_installation_source and
Testcase_Anaconda_traceback_debug_mode all differentiate BIOS and
UEFI. I think that's a waste of our energy. None of these testcases
should IMO be affected by firmware. We should make them generic x86
tests. We could go even further and make them completely
arch-independent (at least some of them), but I'm a bit careful around
ARM, so not completely sure about that.
ACK. Honestly I think this is just because they used to be in the
miscellaneous tests table (some of them still are, I think) and that
thing was freaking huge, and I kinda ran out of steam when I was doing
the revision and just left some of 'em alone).
It's probably worth noting the meaning of a box being available or grey
isn't *entirely* solidly nailed down. It's never been strictly the case
that we require every white box for the appropriate release level to be
filled - back when all we had was i386 and x86_64, we just had a box for
each for every test, and it was kind of a case of 'fill in as many as we
can'. Since we started getting ARM and UEFI it's got a bit more complex
and I think to an extent we're starting to think 'if a box is there it
really needs to be filled'.
In practice when creating and revising the tables (as the person who
does it most of the time) I tend to be slightly fuzzy between 'the boxes
should express the range of all environments for which we really need
this test to happen' or 'the boxes should express the range of
environments in which it's *possible* to do this test, even if we don't
really need to do it in every one of them'. I think we've been trending
towards the former interpretation, though, and that's probably not a bad
idea, for the sake of clarity and reducing the sheer weight of tests.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net