This proposal is for changes to Base test case: "QA:Testcase package install remove" The existing test case is here:
https://fedoraproject.org/wiki/QA:Testcase_package_install_remove
Though there is a requirement to install packages, there are no commands provided to accomplish this. Also, there are no instructions concerning things to look for during the installations.
If we specified the packages to install, we could test an RPM package and a Modular package. I don't think dnf can handle Flatpacks.
This test case uses the command line: (rpm -q package1 package2) This is done twice; once to remove the packages and once to verify the removal.
I might be good to change the first one to: (dnf remove). Though we could use this command again to verify the removal, it might be more revealing to use (dnf info). The DNF info will show, among other things, "Installed" or "Available".
We could even pipe results from the above commands to greps to look for the desired results.
There are many details to be determined. I would like to get feedback on if this test case should be updated and suggestions on the various details I have left open.
Have a Great Day!
Pat tablepc)
On Mon, Feb 3, 2020 at 3:48 PM pmkellly@frontier.com pmkellly@frontier.com wrote:
This proposal is for changes to Base test case: "QA:Testcase package install remove" The existing test case is here:
https://fedoraproject.org/wiki/QA:Testcase_package_install_remove
Though there is a requirement to install packages, there are no commands provided to accomplish this.
The process doesn't need to involve command. The purpose of the test case is to test any package manager, including graphical ones. So for Workstation, we test it with gnome-software. For KDE, we test it with both graphical managers they have in there (because KDE). On Server, we test it with dnf.
Also, there are no instructions concerning things to look for during the installations.
If we specified the packages to install, we could test an RPM package and a Modular package. I don't think dnf can handle Flatpacks.
Modularity has a separate list of test cases, see e.g. here: https://fedoraproject.org/wiki/Test_Results:Fedora_32_Rawhide_20200131.n.0_B...
We have no test cases for flatpaks, at the moment.
This test case uses the command line: (rpm -q package1 package2) This is done twice; once to remove the packages and once to verify the removal.
I might be good to change the first one to: (dnf remove). Though we could use this command again to verify the removal, it might be more revealing to use (dnf info). The DNF info will show, among other things, "Installed" or "Available".
If we want to test dnf, we need to verify the operation with something else, not dnf itself. Rpm is the lowest tool we can use for verification.
On 2/4/20 06:20, Kamil Paral wrote:
On Mon, Feb 3, 2020 at 3:48 PM pmkellly@frontier.com pmkellly@frontier.com wrote:
This proposal is for changes to Base test case: "QA:Testcase package install remove" The existing test case is here:
https://fedoraproject.org/wiki/QA:Testcase_package_install_remove
Though there is a requirement to install packages, there are no commands provided to accomplish this.
The process doesn't need to involve command. The purpose of the test case is to test any package manager, including graphical ones. So for Workstation, we test it with gnome-software. For KDE, we test it with both graphical managers they have in there (because KDE). On Server, we test it with dnf.
There are certainly advantages of being non-specific. You get several package managers tested and likely with several ways of using the package manager. I'm sure someone would install their favorite package manager then run the test.
Also, there are no instructions concerning things to look for during the installations.
If we specified the packages to install, we could test an RPM package and a Modular package. I don't think dnf can handle Flatpacks.
Modularity has a separate list of test cases, see e.g. here: https://fedoraproject.org/wiki/Test_Results:Fedora_32_Rawhide_20200131.n.0_B...
Modularity is part of all of the Fedora composes; is it not? I'm curious why these aren't in the Base cases. Though from what I've seen, there aren't a lot of modules available yet. Is it just because modularity is sort of just getting started. I admit modularity had been off my radar until the Eclipse problem.
We have no test cases for flatpaks, at the moment.
Another one that's just getting started. Though I do have two flatpacks as part of my standard "as deployed" configuration.
This test case uses the command line: (rpm -q package1 package2) This is done twice; once to remove the packages and once to verify the removal.
I might be good to change the first one to: (dnf remove). Though we could use this command again to verify the removal, it might be more revealing to use (dnf info). The DNF info will show, among other things, "Installed" or "Available".
If we want to test dnf, we need to verify the operation with something else, not dnf itself. Rpm is the lowest tool we can use for verification.
Your point is well made. However, since we are letting the testers use their package manager of choice, I would suggest we remove the rpm commands and just instruct them to remove the packages they installed and verify they have been removed.
There is one last point I was thinking of. Though I have never dug into a package manager. I imagine they have some basic test to see that they have installed a package. I'm not sure this is good enough to say that the package actually got installed so that it would work. Perhaps it would be worthwhile to specify one of the two packages (something simple) so the test could also start the application just to see that it would start.
As for removing packages, That's problematic. I know a lot depends on the actual package being removed, but there is always trash left behind. Sometimes other things get broken. As in one of the blocker bugs yesterday. I'm thinking that the only practical test for package removal is that the system still boots and isn't otherwise broken.
Now... This discussion is why I sent the email before just starting on a draft of a modified test case.
Have a Great Day!
Pat (tablepc)
On Tue, Feb 4, 2020 at 3:20 PM pmkellly@frontier.com pmkellly@frontier.com wrote:
On 2/4/20 06:20, Kamil Paral wrote:
On Mon, Feb 3, 2020 at 3:48 PM pmkellly@frontier.com <
pmkellly@frontier.com>
wrote:
This proposal is for changes to Base test case: "QA:Testcase package install remove" The existing test case is here:
https://fedoraproject.org/wiki/QA:Testcase_package_install_remove
Though there is a requirement to install packages, there are no commands provided to accomplish this.
The process doesn't need to involve command. The purpose of the test case is to test any package manager, including graphical ones. So for Workstation, we test it with gnome-software. For KDE, we test it with
both
graphical managers they have in there (because KDE). On Server, we test
it
with dnf.
There are certainly advantages of being non-specific. You get several package managers tested and likely with several ways of using the package manager. I'm sure someone would install their favorite package manager then run the test.
There has to be some misunderstanding here. The purpose is not to install your favorite package manager. This is a release validation test case that is present in the Base matrix and tested against major editions/spins. See here: https://fedoraproject.org/wiki/Test_Results:Fedora_32_Rawhide_20200204.n.0_B...
So for Workstation you use the default one, for KDE the default one(s), for Server the default one, ...
Also, there are no instructions concerning things to look for during the installations.
If we specified the packages to install, we could test an RPM package and a Modular package. I don't think dnf can handle Flatpacks.
Modularity has a separate list of test cases, see e.g. here:
https://fedoraproject.org/wiki/Test_Results:Fedora_32_Rawhide_20200131.n.0_B...
Modularity is part of all of the Fedora composes; is it not? I'm curious why these aren't in the Base cases.
I'm confused. The link I sent you above lists those test cases and they *are* part of the Base matrix, as you can see.
Though from what I've seen, there aren't a lot of modules available yet. Is it just because modularity is sort of just getting started. I admit modularity had been off my radar until the Eclipse problem.
We have no test cases for flatpaks, at the moment.
Another one that's just getting started. Though I do have two flatpacks as part of my standard "as deployed" configuration.
This test case uses the command line: (rpm -q package1 package2) This is done twice; once to remove the packages and once to verify the removal.
I might be good to change the first one to: (dnf remove). Though we could use this command again to verify the removal, it might be more revealing to use (dnf info). The DNF info will show, among other things, "Installed" or "Available".
If we want to test dnf, we need to verify the operation with something else, not dnf itself. Rpm is the lowest tool we can use for verification.
Your point is well made. However, since we are letting the testers use their package manager of choice, I would suggest we remove the rpm commands and just instruct them to remove the packages they installed and verify they have been removed.
I wonder. Have you perhaps found the test case through Bodhi, because (I just noticed) the test case contains these categories? Package PackageKit test casesPackage gnome-software test casesPackage appstream-data test casesPackage libappstream-glib test casesPackage librepo test casesPackage libhif test casesPackage hawkey test casesPackage dnf test casesPackage yum test casesPackage yumex test cases
In that case it the test case shows up in Bodhi for PackageKit/gnome-software/dnf/yum/libhif/hawkey/etc. And it might confuse somebody as to which package manager to use. We can try to clarify that. The obvious answer is the one that you see in Bodhi (packagekit, gnome-software, dnf, etc), but in the case of some of those libraries, it might not be obvious.
There is one last point I was thinking of. Though I have never dug into a package manager. I imagine they have some basic test to see that they have installed a package. I'm not sure this is good enough to say that the package actually got installed so that it would work. Perhaps it would be worthwhile to specify one of the two packages (something simple) so the test could also start the application just to see that it would start.
As for removing packages, That's problematic. I know a lot depends on the actual package being removed, but there is always trash left behind. Sometimes other things get broken. As in one of the blocker bugs yesterday. I'm thinking that the only practical test for package removal is that the system still boots and isn't otherwise broken.
I think that verifying the package manager operation (installation/removal) using `rpm` is completely satisfactory, honestly. Whether the app starts is not really relevant. And whether the system can still boot seems out of scope as well. Of course anything can happen when you run code as root, but the purpose is to check the basics. If the machine bursts into flames, we will notice very quickly and don't need to explicitly check for it in the test case.
A more systematic approach to installation/removal checking would be to check whether the package maintainer only removed the packages you explicitly requested and nothing else. But that quickly gets complex when the user selects packages some other packages depend on, or when the package manager performs some form of "no longer needed packages" checking. This is so complex that it's best automated and not intended as a manual exercise.