Hey All,
Now that we have the F27 Modular Server Beta, It will be great to have the test cases. I've started writing them, I would like to have the feedback and thoughts on them. I have mostly used dnf module --help as a base to derive the these. If something is missing, I would love to draft them too, just let me know.
[1] Listing all modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_module_list [2] Enabling & Disabling modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_enable_disable_modul... [3] Install & Update : https://fedoraproject.org/wiki/User:Sumantrom/Draft/dnf_install_remove_updat... [4] Module Info : https://fedoraproject.org/wiki/User:Sumantrom/Draft/dnf_module_info
Thanks //sumantro
On Thu, Nov 9, 2017 at 8:38 PM Sumantro Mukherjee sumukher@redhat.com wrote:
Hey All,
Now that we have the F27 Modular Server Beta, It will be great to have the test cases. I've started writing them, I would like to have the feedback and thoughts on them. I have mostly used dnf module --help as a base to derive the these. If something is missing, I would love to draft them too, just let me know.
[1] Listing all modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_module_list
Looks good, though we should probably add a set of modules that *must* be enabled/disabled on the default install. It doesn't need to address all possible modules in the list, but we should have a minimum set that must be correct.
[2] Enabling & Disabling modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_enable_disable_modul...
This test doesn't actually validate that the enable/disable correctly happened. At minimum, the results should be tested by repeating [1] and confirming that the values have changed. At best, we should have a set of packages known to install or fail to install that we use to prove the state.
[3] Install & Update : https://fedoraproject.org/wiki/User:Sumantrom/Draft/dnf_install_remove_updat...
This page looks reasonable, but there are numerous typographical errors (including some in the commands) that should be corrected.
It should also add a test for switching streams (e.g. verify that one can install nodejs-8 and then switch it to nodejs-6).
[4] Module Info : https://fedoraproject.org/wiki/User:Sumantrom/Draft/dnf_module_info
This set of test cases needs to have a set of success conditions defined. "It displays some text" isn't good enough for blocker criteria.
Thanks //sumantro _______________________________________________ test mailing list -- test@lists.fedoraproject.org To unsubscribe send an email to test-leave@lists.fedoraproject.org
On Mon, 2017-11-13 at 13:21 +0000, Stephen Gallagher wrote:
On Thu, Nov 9, 2017 at 8:38 PM Sumantro Mukherjee sumukher@redhat.com wrote:
Hey All,
Now that we have the F27 Modular Server Beta, It will be great to have the test cases. I've started writing them, I would like to have the feedback and thoughts on them. I have mostly used dnf module --help as a base to derive the these. If something is missing, I would love to draft them too, just let me know.
[1] Listing all modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_module_list
Looks good, though we should probably add a set of modules that *must* be enabled/disabled on the default install. It doesn't need to address all possible modules in the list, but we should have a minimum set that must be correct.
I'd see that as a separate test. Testing the module list functionality is one thing, defining and checking a set of minimum content for the default Server install is another...
Dear Sumantro,
Did you talk to DNF QE team? I suggest you coordinate your work with Karel Srot from DNF QE team. There is a tool to test DNF and I know Karel already has a good set of tests which covers Modular side of DNF [1], just missing some user stories.
There is also a regular DNF Devs and DNF QE mtg, where we discuss all around DNF QA and particularly Modular DNF, and if you want to contribute you are welcome to join, just please ping Karel all about it.
Also please mind we have a modular DNF BZ tracker [2].
[1] https://github.com/rpm-software-management/ci-dnf-stack/tree/modularity [2] https://bugzilla.redhat.com/show_bug.cgi?id=1478068
Dear Sumantro,
Did you talk to DNF QE team? I suggest you coordinate your work with Karel Srot from DNF QE team. There is a tool to test DNF and I know Karel already has a good set of tests which covers Modular side of DNF [1], just missing some user stories.
We're also trying to keep track of the places where automated testing happens as part of the Fedora module developer's guide at https://docs.pagure.org/modularity/development/building-modules/testing.html
I think the key item we have there that hasn't been mentioned in the thread yet is your own automated tests at https://github.com/fedora-modularity/compose-tests
Cheers, Nick.
P.S. It's entirely plausible that we're missing some aspects of the automated testing setup for the modular server builds, so issue reports and PRs to correct omissions at https://pagure.io/modularity are most welcome.
Nick, thank you for adding this info. I didn't mention it all, since I thought the main intention of Sumantro was to play with DNF, based on posted test cases.
On Tue, 2017-11-14 at 05:04 +0000, Nick Coghlan wrote:
Dear Sumantro,
Did you talk to DNF QE team? I suggest you coordinate your work with Karel Srot from DNF QE team. There is a tool to test DNF and I know Karel already has a good set of tests which covers Modular side of DNF [1], just missing some user stories.
We're also trying to keep track of the places where automated testing happens as part of the Fedora module developer's guide at https://doc s.pagure.org/modularity/development/building-modules/testing.html
I think the key item we have there that hasn't been mentioned in the thread yet is your own automated tests at https://github.com/fedora-m odularity/compose-tests
Cheers, Nick.
P.S. It's entirely plausible that we're missing some aspects of the automated testing setup for the modular server builds, so issue reports and PRs to correct omissions at https://pagure.io/modularity are most welcome.
Note that there is a specific purpose to these test cases: they are for release validation testing. Release validation testing is about ensuring critical functionality works *as experienced by end users*. That is, for release validation testing, we want to test that the most common and critical interactions with modules that will be performed by admins of actual Fedora Server installations work, exactly as those admins would perform them. These tests need to run from clean installs of 'real' composes, and involve using the tools as real-world users will use them. So any tests that involve more artificial environments or interactions are not as useful for release validation purposes. Tests that test "too much" are also not as useful, because we don't immediately know whether a failure is something critical to the release process.
So, we can certainly look at existing tests / tools and see if this testing can share resources with them, but it needs to take the above considerations into account.
On Tue, 2017-11-14 at 13:25 -0800, Adam Williamson wrote:
On Tue, 2017-11-14 at 05:04 +0000, Nick Coghlan wrote:
Dear Sumantro,
Did you talk to DNF QE team? I suggest you coordinate your work with Karel Srot from DNF QE team. There is a tool to test DNF and I know Karel already has a good set of tests which covers Modular side of DNF [1], just missing some user stories.
We're also trying to keep track of the places where automated testing happens as part of the Fedora module developer's guide at https://doc s.pagure.org/modularity/development/building-modules/testing.html
I think the key item we have there that hasn't been mentioned in the thread yet is your own automated tests at https://github.com/fedora-m odularity/compose-tests
Cheers, Nick.
P.S. It's entirely plausible that we're missing some aspects of the automated testing setup for the modular server builds, so issue reports and PRs to correct omissions at https://pagure.io/modularity are most welcome.
Note that there is a specific purpose to these test cases: they are for release validation testing. Release validation testing is about ensuring critical functionality works *as experienced by end users*. That is, for release validation testing, we want to test that the most common and critical interactions with modules that will be performed by admins of actual Fedora Server installations work, exactly as those admins would perform them. These tests need to run from clean installs of 'real' composes, and involve using the tools as real-world users will use them. So any tests that involve more artificial environments or interactions are not as useful for release validation purposes. Tests that test "too much" are also not as useful, because we don't immediately know whether a failure is something critical to the release process.
So, we can certainly look at existing tests / tools and see if this testing can share resources with them, but it needs to take the above considerations into account.
So let's take a concrete example...the ci-dnf-stack tests Irina linked to: https://github.com/rpm-software-management/ci-dnf-stack/tree/modularity
These are great as DNF CI tests, for sure. But they're not as useful for release validation.
Typically, tests at this level want to isolate themselves from external environments. So this test suite contains its own packages and repositories, and the test process uses those. This means it can run in a contained environment and the results will not be affected by changes to packages or repositories that are outside of the test process. Which is exactly what you *want* for this kind of CI testing; you're trying to test that dnf / librepo / hawkey / etc. behaviour is as intended, you don't want external factors to change the results between runs.
However, for distribution-level validation testing, we absolutely *don't* want that. We want to test the actual DNF configuration, repositories and modules we ship with the composes, and we *want to know* if something is broken in those. Just testing that DNF behaves as expected with this set of prefabbed test packages / repositories / modules is not as useful.
On Thu, 2017-11-09 at 20:37 -0500, Sumantro Mukherjee wrote:
Hey All,
Now that we have the F27 Modular Server Beta, It will be great to have the test cases. I've started writing them, I would like to have the feedback and thoughts on them. I have mostly used dnf module --help as a base to derive the these. If something is missing, I would love to draft them too, just let me know.
[1] Listing all modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_module_list [2] Enabling & Disabling modules: https://fedoraproject.org/wiki/User:Sumantrom/Draft/FMS_enable_disable_modul... [3] Install & Update : https://fedoraproject.org/wiki/User:Sumantrom/Draft/dnf_install_remove_updat... [4] Module Info : https://fedoraproject.org/wiki/User:Sumantrom/Draft/dnf_module_info
Along with the suggestions from others, I'd say:
1) Make the setup steps more generic. It is a bad idea to talk about 'Fedora 27' and 'modular server' and even worse to include download links. Think about the philosophy behind the whole release validation design in the wiki: the test cases are made as generic ("vague") as possible. All the context about running the test for a specific release, or image, or whatever, is provided at a different level: a validation event page which specifies the compose it's for and provides download links, for instance, or a Test Day, which similarly specifies what particular image etc. is being tested at that Test Day. We keep that information in the validation event page or the Test Day page, not in the test case, so we don't have to continually update test case pages and duplicate them when we want to run the same test in a different context.
Take a look at the steps in existing post-install validation test cases, for e.g. https://fedoraproject.org/wiki/QA:Testcase_base_update_cli . The text there is a pretty good guide:
"Clean boot the Fedora you wish to test: this could be a system installed from a particular snapshot, pre-release, or release, or a live image. It should be an image for which updates will be available (or you can downgrade a package after installation)."
I'd say to use something like that, but just specify that it must be a *modular* installation.
2) Similarly, the test case names should *not* include 'FMS' or any other reference to modular server specifically; in future, more things than just Server are going to use the Modularity stuff. I'd suggest consistent names something like dnf_module_(something). Perhaps:
dnf_module_list dnf_module_enable_disable dnf_module_install_remove_update dnf_module_info
2) Remove the 'optional' sections - what you've left in is just sample text from the template, don't leave it in if there aren't actually any optional steps. (Also you seem to have some sort of template syntax error in User:Sumantrom/Draft/dnf_module_info ).
3) FMS_enable_disable_module should probably include some kind of verification as to whether the enablement/disablement actually worked.
4) I'm not sure I like the capitalization of Module, Stream, Install Profile etc in dnf_install_remove_update_module . I'd probably put them in italics instead - ''module'', ''stream'', etc.
5) Similarly to 3), the install_remove_update test should include verification that the installation / removal / update was actually performed.
6) It might be worth separating install/remove/update into two tests, one for update, one for install/remove.
Thanks for wokring on this!