Hey, folks. So, as per https://fedorahosted.org/fedora-qa/ticket/307 , I think we need to add a process for validating the multi-live and multi-install images that we've been building for the last few releases. I've put together a draft matrix template for this validation here:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_valid...
It's similar to the other validation matrices, and the changes are mostly obvious. The key bit is the layout of the matrix and the test cases themselves. I haven't actually written any of the specific test cases yet, but I tried to think of the best way to organize things. We obviously need to do the basic sanity tests on each image, then we need to test that each image boots and the menu that lets you select which sub-image to boot works properly, and also that each sub-image boots and installs properly. I think that set of tests ought to pretty much cover the intended functionality of the compendium images. When actually writing the test cases, we might be able to come up with ways to take advantage of some of the existing install/base/desktop validation test cases, but I haven't quite thought that far ahead yet.
I tried to organize the planned test cases as efficiently as possible; this is the best layout I could come up with, but someone might be able to suggest improvements. In case it's not obvious, here's roughly what each test case does:
QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums - these test cases already exist and can apply without modification to the compendium images, they simply check that they're within required size limits (dual-layer DVD size for these images) and the published checksums match the images.
QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will simply check that the compendium images themselves boot, when written to a physical dual-layer DVD and booted as normal. We _may_ also want to test booting them when written to USB using one specific method. I don't think we need to test any other way of booting the images, as these images are specifically intended to be written to physical media and given out at events; we don't really intend people to download them and write/boot them via all possible methods themselves, though they can do that if they really want to. So we don't really need to cover VM booting of the image, writing to USB via all other possible methods and so on.
QA:Testcase_multilive_menu and QA:Testcase_multidvd_menu - these will test that the top-level menus on each image, for picking which sub-image to boot, function correctly. I'm not sure at present if it'll be possible to just instead write a generic QA:Testcase_multi_menu test case which would apply to either image; if so, we can do just QA:Testcase_multi_menu and QA:Testcase_multi_boot test cases and have one result column for each image (as in the Image sanity table) instead of four test cases and a single result column.
QA:Testcase_multi_sub_boot and QA:Testcase_multi_sub_install - these will test that each sub-image on each compendium image boots and installs correctly. It would probably be unusual for _just one_ sub-image to fail, so we could probably get away with just testing a single sub-image on each medium, but everything that can go wrong will go wrong, so we might as well cover everything if we have time/resources. We may be able to re-use an existing test case instead of writing new ones here, I'll have to check the exact wording of some of our existing cases.
Thoughts, comments, objections, suggestions? Can everyone see where this is going, or am I zooming off up the wrong tree? Can you think of a better approach? Thanks!
Hey, folks. So, as per https://fedorahosted.org/fedora-qa/ticket/307 , I think we need to add a process for validating the multi-live and multi-install images that we've been building for the last few releases.
It is ironic that before F17 release I have talked about this to cwickert (IIRC) and he assured me no QA help is needed and that nothing can go wrong. When they build the image, they just ensure everything boots and that's it. There's just a small syslinux glue, nothing else is changed. Well, it didn't go according to the plan, apparently.
I've put together a draft matrix template for this validation here:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_valid...
It's similar to the other validation matrices, and the changes are mostly obvious. The key bit is the layout of the matrix and the test cases themselves. I haven't actually written any of the specific test cases yet, but I tried to think of the best way to organize things. We obviously need to do the basic sanity tests on each image, then we need to test that each image boots and the menu that lets you select which sub-image to boot works properly, and also that each sub-image boots and installs properly. I think that set of tests ought to pretty much cover the intended functionality of the compendium images. When actually writing the test cases, we might be able to come up with ways to take advantage of some of the existing install/base/desktop validation test cases, but I haven't quite thought that far ahead yet.
I tried to organize the planned test cases as efficiently as possible; this is the best layout I could come up with, but someone might be able to suggest improvements. In case it's not obvious, here's roughly what each test case does:
QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums
these test cases already exist and can apply without modification to the compendium images, they simply check that they're within required size limits (dual-layer DVD size for these images) and the published checksums match the images.
QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will simply check that the compendium images themselves boot
I don't understand what this means.
To clarify, multidvd structure is this: -> default DVD boot (auto-selects architecture) -> select arch manually -> i386 -> x86_64 -> basic video -> both 2 options -> memory test -> local drive boot
So you probably mean the "default DVD boot" here, right? But which architecture? The result might differ on i386 and on x86_64 machine.
But multilive structure is this: -> default Live GNOME (auto-selects architecture) -> default Live KDE (auto-selects architecture) -> default Live LXDE (auto-selects architecture) -> default Live SoaS (auto-selects architecture) -> default Live XFCE (auto-selects architecture) -> select arch manually -> i386 -> Live GNOME -> Live KDE -> Live LXDE -> Live SoaS -> Live XFCE -> x86_64 -> Live GNOME -> Live KDE -> Live LXDE -> Live SoaS -> Live XFCE -> basic video -> all 10 options here -> memory test -> local drive boot
What does the test refer to for multilive?
when written to a physical dual-layer DVD and booted as normal.
How important is here to really burn it? Is ISO VM booting regarded as a reliable test or not? (Note: We don't burn classic Fedora images when testing either.)
We _may_ also want to test booting them when written to USB using one specific method.
I don't think so. This is just to be pressed to DVDs. If Red Hat ever starts to give away USB sticks, we can reconsider. Until then, it's redundant work.
I don't think we need to test any other way of booting the images, as these images are specifically intended to be written to physical media and given out at events; we don't really intend people to download them and write/boot them via all possible methods themselves, though they can do that if they really want to. So we don't really need to cover VM booting of the image, writing to USB via all other possible methods and so on.
QA:Testcase_multilive_menu and QA:Testcase_multidvd_menu - these will test that the top-level menus on each image, for picking which sub-image to boot, function correctly. I'm not sure at present if it'll be possible to just instead write a generic QA:Testcase_multi_menu test case which would apply to either image; if so, we can do just QA:Testcase_multi_menu and QA:Testcase_multi_boot test cases and have one result column for each image (as in the Image sanity table) instead of four test cases and a single result column.
QA:Testcase_multi_sub_boot and QA:Testcase_multi_sub_install - these will test that each sub-image on each compendium image boots and installs correctly. It would probably be unusual for _just one_ sub-image to fail, so we could probably get away with just testing a single sub-image on each medium, but everything that can go wrong will go wrong, so we might as well cover everything if we have time/resources. We may be able to re-use an existing test case instead of writing new ones here, I'll have to check the exact wording of some of our existing cases.
Thoughts, comments, objections, suggestions? Can everyone see where this is going, or am I zooming off up the wrong tree? Can you think of a better approach? Thanks!
I have to say I'm somewhat confused from the current matrices structure. But maybe when the test cases are written it'll clearer.
My idea personally was:
MultiLive matrix | GNOME | KDE | LXDE | SoaS | XFCE ------------------------------------------------------------------ multilive_auto_boot (i386) | multilive_auto_boot (x86_64) | multilive_manual_boot (i386) | multilive_manual_boot (x86_64) | multilive_install (anyarch) |
"auto_boot" is when you select the default option and correct architecture is chosen for you. To test this properly, you need both i386 and x86_64 machines (or VMs, if accepted). "manual_boot" is when you select the architecture manually from the submenu. A single x86_64 machine is sufficient to check both cases.
MultiDVD matrix | result --------------------------------------- multidvd_auto_boot (i386) | multidvd_auto_boot (x86_64) | multidvd_manual_boot (i386) | multidvd_manual_boot (x86_64) | multidvd_install (anyarch) |
I am wondering whether we need installation tests at all. Theoretically the ISOs are unchanged so there should be no difference at all. If they start to boot, it should be fine. But according to Murphy's Law...
On Thu, 2012-08-09 at 05:49 -0400, Kamil Paral wrote:
QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will simply check that the compendium images themselves boot
I don't understand what this means.
To clarify, multidvd structure is this: -> default DVD boot (auto-selects architecture) -> select arch manually -> i386 -> x86_64 -> basic video -> both 2 options -> memory test -> local drive boot
So you probably mean the "default DVD boot" here, right?
No. I simply mean 'do you get to the sub-image selection menu'.
when written to a physical dual-layer DVD and booted as normal.
How important is here to really burn it? Is ISO VM booting regarded as a reliable test or not? (Note: We don't burn classic Fedora images when testing either.)
I would want to, yes. I do burn classic images when testing; the tests https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Dvd and https://fedoraproject.org/wiki/QA:Testcase_Live_Image_Boot specifically require writing the images to media. (They currently refer to USB as well as optical media, but I'm going to propose changing that, because we have specific tests for USB now; those tests are intended to cover precisely the case of actually booting the images from the optical media to which they were originally intended to be written).
_In theory_ booting a VM which is treating the image file as an optical disc always ought to be good enough for testing, but we all know how well theory works when it comes to QA =)
We _may_ also want to test booting them when written to USB using one specific method.
I don't think so. This is just to be pressed to DVDs. If Red Hat ever starts to give away USB sticks, we can reconsider. Until then, it's redundant work.
Right, that was the scenario I was considering; if there aren't any immediate plans to start distributing USB sticks, we can skip it.
"auto_boot" is when you select the default option and correct architecture is chosen for you. To test this properly, you need both i386 and x86_64 machines (or VMs, if accepted).
For the sub-image boot testing I think using a VM is okay. I hadn't considered the automatic arch selection wrinkle, I'll think up a way to adjust for that - thanks.
I am wondering whether we need installation tests at all. Theoretically the ISOs are unchanged so there should be no difference at all. If they start to boot, it should be fine. But according to Murphy's Law...
Exactly. _Theoretically_, you're correct. ;)
How important is here to really burn it? Is ISO VM booting regarded as a reliable test or not? (Note: We don't burn classic Fedora images when testing either.)
I would want to, yes. I do burn classic images when testing; the tests https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Dvd and https://fedoraproject.org/wiki/QA:Testcase_Live_Image_Boot specifically require writing the images to media.
Oops, and I was doing it wrong all the time.
On Fri, 2012-08-10 at 03:23 -0400, Kamil Paral wrote:
How important is here to really burn it? Is ISO VM booting regarded as a reliable test or not? (Note: We don't burn classic Fedora images when testing either.)
I would want to, yes. I do burn classic images when testing; the tests https://fedoraproject.org/wiki/QA:Testcase_Boot_Methods_Dvd and https://fedoraproject.org/wiki/QA:Testcase_Live_Image_Boot specifically require writing the images to media.
Oops, and I was doing it wrong all the time.
Welp, robatino says he always read it the same way you did, so it's definitely not clear. I think my reading actually reflects the initial intent of the tests, but we could definitely clarify things.
So, this is how I think it ought to work:
* the boot_iso, boot_methods_dvd, and live_image_boot tests should specifically test booting of an actual physical optical disc (we should explain this more clearly, and remove the USB references now we have dedicated USB tests)
* the 'USB stick' table tests should specifically test booting of physical USB media (this is already clear, so that's fine)
* the current_KVM and previous_KVM tests (and Xen_Para_Virt, for that matter) cover booting of the images attached to VMs (not really in as much depth, but in practice I think it's fine, as we do lots of testing in KVMs anyhow)
* all the other 'general tests' are tests of installer functionality, and can be done from any boot method
Is this general scheme OK with everyone? If so, I'll propose a few test revisions to make it much more clear. Thanks!
On Thu, 2012-08-09 at 05:49 -0400, Kamil Paral wrote:
Hey, folks. So, as per https://fedorahosted.org/fedora-qa/ticket/307 , I think we need to add a process for validating the multi-live and multi-install images that we've been building for the last few releases.
It is ironic that before F17 release I have talked about this to cwickert (IIRC) and he assured me no QA help is needed and that nothing can go wrong. When they build the image, they just ensure everything boots and that's it. There's just a small syslinux glue, nothing else is changed. Well, it didn't go according to the plan, apparently.
I've put together a draft matrix template for this validation here:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_valid...
It's similar to the other validation matrices, and the changes are mostly obvious. The key bit is the layout of the matrix and the test cases themselves. I haven't actually written any of the specific test cases yet, but I tried to think of the best way to organize things. We obviously need to do the basic sanity tests on each image, then we need to test that each image boots and the menu that lets you select which sub-image to boot works properly, and also that each sub-image boots and installs properly. I think that set of tests ought to pretty much cover the intended functionality of the compendium images. When actually writing the test cases, we might be able to come up with ways to take advantage of some of the existing install/base/desktop validation test cases, but I haven't quite thought that far ahead yet.
I tried to organize the planned test cases as efficiently as possible; this is the best layout I could come up with, but someone might be able to suggest improvements. In case it's not obvious, here's roughly what each test case does:
QA:Testcase Mediakit_ISO_Size and QA:Testcase_Mediakit_ISO_Checksums
these test cases already exist and can apply without modification to the compendium images, they simply check that they're within required size limits (dual-layer DVD size for these images) and the published checksums match the images.
QA:Testcase_multilive_boot and QA:Testcase_multidvd_boot - these will simply check that the compendium images themselves boot
I don't understand what this means.
To clarify, multidvd structure is this: -> default DVD boot (auto-selects architecture) -> select arch manually -> i386 -> x86_64 -> basic video -> both 2 options -> memory test -> local drive boot
So you probably mean the "default DVD boot" here, right? But which architecture? The result might differ on i386 and on x86_64 machine.
But multilive structure is this: -> default Live GNOME (auto-selects architecture) -> default Live KDE (auto-selects architecture) -> default Live LXDE (auto-selects architecture) -> default Live SoaS (auto-selects architecture) -> default Live XFCE (auto-selects architecture) -> select arch manually -> i386 -> Live GNOME -> Live KDE -> Live LXDE -> Live SoaS -> Live XFCE -> x86_64 -> Live GNOME -> Live KDE -> Live LXDE -> Live SoaS -> Live XFCE -> basic video -> all 10 options here -> memory test -> local drive boot
What does the test refer to for multilive?
I have to say I'm somewhat confused from the current matrices structure. But maybe when the test cases are written it'll clearer.
My idea personally was:
MultiLive matrix | GNOME | KDE | LXDE | SoaS | XFCE
multilive_auto_boot (i386) | multilive_auto_boot (x86_64) | multilive_manual_boot (i386) | multilive_manual_boot (x86_64) | multilive_install (anyarch) |
"auto_boot" is when you select the default option and correct architecture is chosen for you. To test this properly, you need both i386 and x86_64 machines (or VMs, if accepted). "manual_boot" is when you select the architecture manually from the submenu. A single x86_64 machine is sufficient to check both cases.
MultiDVD matrix | result
multidvd_auto_boot (i386) | multidvd_auto_boot (x86_64) | multidvd_manual_boot (i386) | multidvd_manual_boot (x86_64) | multidvd_install (anyarch) |
I am wondering whether we need installation tests at all. Theoretically the ISOs are unchanged so there should be no difference at all. If they start to boot, it should be fine. But according to Murphy's Law...
OK, I revised the middle table somewhat, I tried to figure it out myself but it turns out fairly similar to yours:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_valid...
so 'multi_auto' would just be 'leave it, and check it does the right thing', 'multi_manual' would be a test case that's something like 'try every entry besides basic video / memtest / local boot and make sure they respond appropriately', and the other three are obvious - we probably don't really need to do those three on both arches, we could split out a small table for those, I guess.
The multi_sub_boot test now becomes redundant, and I suppose we could move sub_install up into this new big table; we can probably get away without testing install of every DE, since they all use anaconda. The case I can see where install might break on the compendium images is the case where anaconda, for whatever reason, examines the actual medium it's booting from. It _does_ do this, in some cases - like it tries to except the USB stick it's booting from (if it is) from the target device list. (This test may well not work on the compendium images, actually). I can possibly imagine a case where anaconda goes to look what image it's booting from, and gets confused if it's the compendium disc. But that shouldn't vary between desktops.
This is a pretty tricky problem space and I'm not at all confident I'm wrapping my head around it perfectly, more comments and improvements of course welcome =) Thanks!
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_valid...
so 'multi_auto' would just be 'leave it, and check it does the right thing', 'multi_manual' would be a test case that's something like 'try every entry besides basic video / memtest / local boot and make sure they respond appropriately', and the other three are obvious - we probably don't really need to do those three on both arches, we could split out a small table for those, I guess.
This table is pretty neat.
One thing though: I find "Multi-Live" and "Multi-DVD" names more straightforward than "Multi-Desktop" and "Multi-Install" (this one takes me a while to decipher). But that might be my personal feeling only.
The multi_sub_boot test now becomes redundant,
Yes, that's not needed now.
and I suppose we could move sub_install up into this new big table; we can probably get away without testing install of every DE, since they all use anaconda. The case I can see where install might break on the compendium images is the case where anaconda, for whatever reason, examines the actual medium it's booting from. It _does_ do this, in some cases - like it tries to except the USB stick it's booting from (if it is) from the target device list. (This test may well not work on the compendium images, actually). I can possibly imagine a case where anaconda goes to look what image it's booting from, and gets confused if it's the compendium disc. But that shouldn't vary between desktops.
I see it the same way. That means multi_sub_install (or maybe just multi_install now) can be merged into the previous table.
On 2012-08-13 5:39, Kamil Paral wrote:
https://fedoraproject.org/wiki/User:Adamwill/Draft_QA_Compendium_image_valid...
so 'multi_auto' would just be 'leave it, and check it does the right thing', 'multi_manual' would be a test case that's something like 'try every entry besides basic video / memtest / local boot and make sure they respond appropriately', and the other three are obvious - we probably don't really need to do those three on both arches, we could split out a small table for those, I guess.
This table is pretty neat.
One thing though: I find "Multi-Live" and "Multi-DVD" names more straightforward than "Multi-Desktop" and "Multi-Install" (this one takes me a while to decipher). But that might be my personal feeling only.
Glad we're on the same wavelength now! Anyone else have comments?
On the names: I used the same names from the Testcase_Mediakit_ISO_Checksums test case, for consistency. But I'm not sure if we have 'official' names for these, really. https://fedoraproject.org/wiki/Artwork/MediaArt/F17 isn't terribly consistent about what it calls them. https://fedoraproject.org/wiki/Multi_Boot_Media_SOP goes with 'Multi Desktop DVD' and 'Dual Arch Installer DVD'. Maybe we need to come up with something official :)