So after some further consideration I think I'm on board with
this, and
it turned out not to be too hard to implement. I have updated the draft
test case:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Testcase_Boot_default_...
so it now basically says 'you can run any kind of install that boots
from the medium under test and uses the default repo, and if it
satisfies the Expected Results, we're good'.
Does this look better to you now?
Yes, thanks.
I think this is going a bit too far. I'm OK with not requiring
complete
coverage at Alpha or Beta, but I think explicitly moving optical boot
to the Final criteria is too much. I think if we do find out that
optical boot is broken for a release-blocking medium at Alpha or Beta
stage that should still be a blocker. At least for now, until the
project as a whole decides we don't care about optical media any more.
OK. However, unless you're strongly opposed, I'd like to start discussing exactly
this (requiring optical media to work only for Final, not before) and have it agreed on
for F26. From my POV, there's little gain in making sure optical works for Alpha/Beta.
Optical is important for conservative audience with very old machines, and is probably in
a similar position to i386. Also for giveaways. All of that is important for Final, but
isn't really for Alpha and Beta. I don't think we have too many users which
regularly install the latest and greatest betaversions from optical drives. (Also, the
risk of breaking just optical boot on bare metal seems pretty low, even though it happened
in the past).
So I went with a halfway house. I've updated the matrix draft too:
https://fedoraproject.org/wiki/User:Adamwill/Draft_Installation_test_matrix
so it now lists the milestone as 'Alpha / Final' and has an admon that
explicitly explains the 'Expected coverage' for the section, saying we
expect a reasonable sample of tests for Alpha and Beta, and full
coverage for Final. Do you think that's sufficient?
Just to have an idea, say that we have both VM* columns fully covered by OpenQA, but for
DVD columns with have just a single BIOS+UEFI result for Workstation Live. Would that be
sufficient for Beta? What do we consider "a reasonable coverage"?
Note that with the current wikitcms code, testcase_stats considers the
milestone for 'Alpha / Final' rows to be 'Final'...because it works
like this:
for mile in ('Alpha', 'Beta', 'Final',
'Optional', 'Tier1', 'Tier2',
'Tier3'):
if mile in cells[0]:
milestone = mile
obviously that would be very trivial to tweak, I just thought I'd
mention it. I might tweak it to prefer the first milestone it finds
rather than the last, I'll just have to look and see if that would
cause any unexpected effects in other cases (I don't think we have
many, though).
Yeah, tweaking that would probably make sense.