> 1. Testcase_Anaconda_User_Interface_Basic_Video_Driver was
changed to
> a generic x86 test. In this case, it is problematic, because BIOS and
> UEFI fallback video driver are completely different code paths. I have
> a personal experience from that past when one of those worked and the
> other didn't. We should distinguish BIOS and UEFI in this case. To fix
> this, I see two viable options:
> a) move the test case to Miscellaneous section, so that the table in
> User interface section doesn't get more complicated
> b) keep the test case in User interface section, but create a separate
> table for it (right under the first table). This new table would have
> x86 BIOS and x86 UEFI columns. Again, the purpose is to keep the
> original table simple (avoid adding columns which are not required for
> anything else).
The tests in that table are really covering somewhat different things,
now I look at them. Graphical, and Text actually exercise two of
anaconda's UIs (we seem to be missing a test for cmdline, unless one of
the other test cases hits it in passing). The others are different ways
of accessing those same UIs: the VNC tests are different ways to get at
graphical, and serial_console is another way to get at text.
It makes sense for Graphical and Text to not be split between i686,
x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to
have probably x86 BIOS / x86 UEFI split, indeed. We should consider the
other tests in deciding the right thing to do - what's the appropriate
split for VNC and serial console?
For VNC, we require the vnc binary (and networking) to work correctly. Of course there can
be arch-specific bugs. But that's true for almost anything, we're working with
likelihood-to-break-in-just-a-single-arch estimates here. From that POV, I think the test
case is very similar to NFS/NFSISO repo testcase - again, you need the NFS binaries to
work correctly. And we have just a single Result column for that.
As I said, I don't feel well acquainted with ARM to guess which areas are more or less
likely to work differently than on x86, and thus likely to contain arch-specific bugs. But
I'd either keep current x86 & ARM separation, or unify that into a single test
result, as with NFS.
Serial console is likely to be more different between x86 and ARM than VNC (I seem to
remember some ARM specific serial console bugs from this cycle), so the current separation
seems reasonably there.
Another question is whether we should keep the same approach for Text interface test case,
or whether that can be united into a single result. Not sure here.
> 2. Testcase_Install_to_Previous_KVM and
> Testcase_Install_to_Current_KVM differentiate BIOS and UEFI. I always
> forget whether this is about the host or the guest (or both, or which
> combination). But I think host differentiation doesn't make any
> difference - libvirt should work the same, I assume. It makes sense to
> differentiate the guests. Unfortunately I believe the UEFI mode for
> libvirt/kvm [2] is not available under an open source license,
> therefore it's not packaged and not trivial to use. Which means not
> that many people probably use it. I wonder whether we really need to
> keep this separated. I would gladly merge this together to a single
> x86 column and split up once OVMF is available under a better license.
> If we keep it separate, we should provide better instructions for the
> UEFI part.
Well, the licensing issue is a bit subtle. OVMF itself is freely
licensed (BSD or MIT IIRC). The bit that has a license issue is the FAT
driver used by EDK II, on which OVMF is built. It has this extra term in
its license:
Additional terms: In addition to the forgoing, redistribution and use
of the code is conditioned upon the FAT 32 File System Driver and all
derivative works thereof being used for and designed only to read
and/or write to a file system that is directly managed by Intel's
Extensible Firmware Initiative (EFI) Specification v. 1.0 and later
and/or the Unified Extensible Firmware Interface (UEFI) Forum's UEFI
Specifications v.2.0 and later (together the "UEFI Specifications");
only as necessary to emulate an implementation of the UEFI
Specifications;
and to create firmware, applications, utilities and/or drivers.
which obviously makes it non-free. I believe the *reason* is something
to do with Microsoft patent(s) believed to be related to FAT, and I
don't think it's prudent to say anything more specific than that.
There therefore isn't a legal issue with redistributing OVMF builds, and
there is a commonly-used repository which contains them, updated daily:
https://www.kraxel.org/repos/
they just can't go into Fedora, because of Fedora's policies about
non-freely-licensed software. It's pretty easy to set up a UEFI KVM
using the packages from that repo, I have one here and just spent all
morning using it to test UEFI multi-boot.
We certainly *can* block Fedora releases on things that have to do with
software that's not part of Fedora's repositories, if we want to - viz
the Windows or OS X boot criteria, for e.g. It'd be interesting to see,
if a blocker was proposed, if we actually want to block on UEFI KVM
guest operation, but I think at least it's worth having it in the
matrix.
Thanks for explanation, I adjusted the test cases a bit:
https://fedoraproject.org/w/index.php?title=QA%3ATestcase_Install_to_Prev...
https://fedoraproject.org/w/index.php?title=QA%3ATestcase_Install_to_Curr...
> 3. Testcase_Anaconda_save_traceback_to_bugzilla,
> Testcase_Anaconda_updates.img_via_URL,
> Testcase_Anaconda_user_creation,
> Testcase_Anaconda_updates.img_via_installation_source and
> Testcase_Anaconda_traceback_debug_mode all differentiate BIOS and
> UEFI. I think that's a waste of our energy. None of these testcases
> should IMO be affected by firmware. We should make them generic x86
> tests. We could go even further and make them completely
> arch-independent (at least some of them), but I'm a bit careful around
> ARM, so not completely sure about that.
ACK. Honestly I think this is just because they used to be in the
miscellaneous tests table (some of them still are, I think) and that
thing was freaking huge, and I kinda ran out of steam when I was doing
the revision and just left some of 'em alone).
All of them are from Miscellaneous section. Do you have a good idea how to make then x86
generic without making the tables too complex?
I can add a fourth 'x86' column, and gray out BIOS and UEFI columns for these
discussed test cases. Which is usable, but not completely nice (the table source code
grows a lot).
Or I can create a separate table in this section. I think this is nicer and we'll end
up doing this in some sections in the future anyway. Do you see some disadvantage in it?
It's probably worth noting the meaning of a box being available or grey
isn't *entirely* solidly nailed down. It's never been strictly the case
that we require every white box for the appropriate release level to be
filled - back when all we had was i386 and x86_64, we just had a box for
each for every test, and it was kind of a case of 'fill in as many as we
can'. Since we started getting ARM and UEFI it's got a bit more complex
and I think to an extent we're starting to think 'if a box is there it
really needs to be filled'.
In practice when creating and revising the tables (as the person who
does it most of the time) I tend to be slightly fuzzy between 'the boxes
should express the range of all environments for which we really need
this test to happen' or 'the boxes should express the range of
environments in which it's *possible* to do this test, even if we don't
really need to do it in every one of them'. I think we've been trending
towards the former interpretation, though, and that's probably not a bad
idea, for the sake of clarity and reducing the sheer weight of tests.
Yes, this is exactly the reason why I try to reduce the number of boxes. Because even
though in theory it would be nice to have fields for every possible combination and people
would fill them according to the real priority (e.g. it's better to test 2 different
test cases on a single arch, than a single test case on 2 arches), in practice often the
opposite happens. We don't have this priority specified anywhere ("is it really
important to test run test case on all arches or not?"), and even if we did, it would
probably make the matrix horribly complicated. So people tend to do the opposite approach
- I have read this test case, I remember it now, so let's run it for all architectures
available, and only then continue to the next one. I do that often as well, even though
it's not efficient from test coverage POV, because it's much easier especially
when you're juggling with several tasks at the same time. And it's also easier
than to try to decide which white boxes are mandatory and which ones are optional.
The second thing is that we don't usually have too much spare time for running extra
test cases anyway, so it's better to optimize the required path as much as possible,
and if people end up with spare time, usually they find more bugs when they're just
playing with software and experimenting, than testing the same stuff on a bit different
hardware.