> The new matrix version seems to be more time consuming that the
> previous version.
In fact, the count of 'empty spaces' is identical in both versions. In
the old version there were 24 empty spaces that humans were *supposed*
to be filling in (though this wasn't always really happening): 12 in
the "Default boot and install" table, and 12 in the USB table (it's a
4x3 table). In the new version, there are...24 empty spaces for humans
to fill in in the "Default boot and install (x86_64)" table.
Yes, their number is the same, but I meant the amount of time required. The USB test cases
allowed us to combine them with other test cases (e.g. partitioning) or perform a minimal
install, while the new test case doesn't (a default install is mandatory).
> a) could we decouple boot from install?
>
> I know we've had a few problems in the past that occurred only on
> bare metal and not in VMs. But do I remember correctly that all of
> those were related to booting the installer/live system, and not the
> installation? What we do here is that we repeat the default
> installation over and over again, e.g. Workstation Live using BIOS
> and UEFI (which results in a slightly different partition layout),
> from different sources (DVD in VM, DVD on bare metal, USB). But do
> those environments (VM vs bare metal, DVD vs USB) actually matter, is
> there a good reason to repeat them that many times? What if we reduce
> all of this to:
>
> Default install
> ===============
> Result
> Workstation Live [ ]
> Workstation netinst [ ]
> Everything netinst [ ]
> Server netinst [ ]
> Server DVD [ ]
> KDE Live [ ]
>
> which would be satisfiable using ANY environment and it would just
> check that anaconda can use that particular installation source
> correctly? The BIOS vs UEFI partitioning is already covered
> thoroughly in the "Guided storage configuration" table.
> The currently proposed "Default boot and install" section would stay
> the same, but be renamed to "Default boot" and would just check that
> the image *boots* into the installer in that particular environment.
I think this is going a bit too far. There are, IIRC, at least two
known places beyond just initial boot where things can go wrong in ways
related to the install medium.
One is install target selection. anaconda has code for filtering out
'protected' devices which is intended (in part) to prevent you trying
to install to the USB stick you're running the install from. That code
can go wrong in various ways, including over-aggressive filtering, and
the bugs can be media-specific (e.g. there's no problem in a VM or from
a real optical disc, but it goes wrong when booting from a USB stick).
The other is use of the on-medium repository. This is mainly for the
Server DVD image. When you install from that image the install should
use the packages from the medium itself as the base repository (and not
go out and get them from the network); I believe we've seen a case
before where the Server DVD written to USB would boot OK, and even
install OK, but it didn't actually use the USB stick as a repository,
it acted like a network install image. When written to a DVD it worked
OK. That was a long time ago, though, and I don't have the bug
reference handy.
This is why the test case has this text:
"If the image under test is intended to act as a package source, it
must be the default repository and installation must in fact use the
media as the package source."
OK, you're right that there are some cases which could bypass our testing. That's
why I assumed this idea might not fly, even though I'd personally have no problem to
focus on "reasonable coverage" instead of full coverage. But there might be
better approaches to handle this than my "a)" proposal.
From all that you described above, it seems that none of the important features to be
tested should be negatively affected if we allow arbitrary partitioning or a different
package set (provided the default installation source is used). Or even other, like VNC or
basic video mode. What needs to stay fixed is just the install media + environment +
install source. Which essentially my "b)" proposal in the original email. That
would allow us to be more effective when going through "Default boot and
install" table while (hopefully) not sacrificing any test coverage. What do you think
about that?
We *could* reduce the test case to 'boot, select a target disk and
start the installation process if using a DVD image', I guess. Would
this sufficiently address your concerns?
That would definitely speed things up. But it feels to me a bit ugly to kill installations
during execution as an official test approach. Would the idea written above serve better?
It's not that fast and easy (the installations should still finish and you should
verify the system boots), but since it can be combined with other test cases, it should
not effectively slow us down (too much).
Of course it's *possible* that some weird bug could show up which made
bootloader configuration not work properly in some media-dependent way,
or something. But I don't recall us running into such a bug before.
> >
> > At present I'm not proposing any changes to the *release criteria* (we
> > can maybe consider that later). This is just a proposal for what we
> > will actually guarantee testing of via the validation process.
>
> If we go full steam ahead with optical disk testing as proposed (in
> the past, the "default boot and install" test case said you can
> choose either USB or DVD, now we have separate fields for both
> implying we need to test both) I'd suggest optical media being only
> in the Final criteria.
That was my intent, but it's a bit hard to express with the way we lay
out the wiki tables, unfortunately (hard to specify different
milestones for different environments). I can add an admon note above
the table, I guess, explaining precisely what's actually required for
each milestone. How does that sound?
I'd put "Alpha/Final" into the Milestone in the "Default boot and
install" table. And the testcase would contain an admon bar linking to a Final
criterion saying "installation must work from optical media". That could be
obvious enough (at least for us, initiated testers) and follow the same pattern we use for
other such mixed-milestone test cases. Wdyt?