On Mon, 2014-11-10 at 05:13 -0500, Kamil Paral wrote:
> It makes sense for Graphical and Text to not be split between
i686,
> x86_64 and UEFI, for sure. And it makes sense for Basic Video Driver to
> have probably x86 BIOS / x86 UEFI split, indeed. We should consider the
> other tests in deciding the right thing to do - what's the appropriate
> split for VNC and serial console?
For VNC, we require the vnc binary (and networking) to work correctly.
Of course there can be arch-specific bugs. But that's true for almost
anything, we're working with likelihood-to-break-in-just-a-single-arch
estimates here. From that POV, I think the test case is very similar
to NFS/NFSISO repo testcase - again, you need the NFS binaries to work
correctly. And we have just a single Result column for that.
As I said, I don't feel well acquainted with ARM to guess which areas
are more or less likely to work differently than on x86, and thus
likely to contain arch-specific bugs. But I'd either keep current x86
& ARM separation, or unify that into a single test result, as with
NFS.
Serial console is likely to be more different between x86 and ARM than
VNC (I seem to remember some ARM specific serial console bugs from
this cycle), so the current separation seems reasonably there.
Another question is whether we should keep the same approach for Text
interface test case, or whether that can be united into a single
result. Not sure here.
If you want my guess, I guess I'd suggest we make VNC non-arch-specific,
keep serial console split ARM/x86, and make text non-interface-specific,
it seems like as long as you get a console at all, that ought to work.
> ACK. Honestly I think this is just because they used to be in
the
> miscellaneous tests table (some of them still are, I think) and that
> thing was freaking huge, and I kinda ran out of steam when I was doing
> the revision and just left some of 'em alone).
All of them are from Miscellaneous section. Do you have a good idea
how to make then x86 generic without making the tables too complex?
I can add a fourth 'x86' column, and gray out BIOS and UEFI columns
for these discussed test cases. Which is usable, but not completely
nice (the table source code grows a lot).
Or I can create a separate table in this section. I think this is
nicer and we'll end up doing this in some sections in the future
anyway. Do you see some disadvantage in it?
No, I agree, I was kinda working to split the misc section up. It fits
the design most nicely if you can come up with some sort of 'topic'
connection between the cases that live together in a table, but the
*functional* reason to have split tables is indeed to group tests with
similar result environments, and I don't like having too much greying
out of the rows either.
We should really limit the greying-out hack and use it only when a test
case is *clearly* part of a set with a lot of others but the
environments that otherwise apply to them all don't apply to it - i.e.
it should be a 'rare exception' case, we shouldn't be building complex
sets of result columns where each one might have only one or two white
boxes.
Yes, this is exactly the reason why I try to reduce the number of
boxes. Because even though in theory it would be nice to have fields
for every possible combination and people would fill them according to
the real priority (e.g. it's better to test 2 different test cases on
a single arch, than a single test case on 2 arches), in practice often
the opposite happens. We don't have this priority specified anywhere
("is it really important to test run test case on all arches or
not?"), and even if we did, it would probably make the matrix horribly
complicated. So people tend to do the opposite approach - I have read
this test case, I remember it now, so let's run it for all
architectures available, and only then continue to the next one. I do
that often as well, even though it's not efficient from test coverage
POV, because it's much easier especially when you're juggling with
several tasks at the same time. And it's also easier than to try to
decide which white boxes are mandatory and which ones are optional.
The second thing is that we don't usually have too much spare time for
running extra test cases anyway, so it's better to optimize the
required path as much as possible, and if people end up with spare
time, usually they find more bugs when they're just playing with
software and experimenting, than testing the same stuff on a bit
different hardware.
Yeah, I more or less agree entirely with all of this. You can count me
generally in favour of any change which rationalizes the tables on the
basis of moving more towards 'white boxes express required testing'.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net