When a critical path sits for any more than four weeks I suggest the maintainer include test files and test cases to help any tester. Things such as scripts/programs to test with, or scenarios like do this and check that.
On Sun, Oct 10, 2010 at 18:49:10 -0400, Timothy Davis cpuobsessed@gmail.com wrote:
When a critical path sits for any more than four weeks I suggest the maintainer include test files and test cases to help any tester. Things such as scripts/programs to test with, or scenarios like do this and check that.
For example for squashfs, the bug was hard to test. Kevin saw it, but I didn't. We weren't sure exactly what changed that started breaking things for the nightly composes that Kevin is doing. I was more interested in making sure there weren't regressions than making sure that the bug was really fixed.
Having automated tests or even test plans would be useful for any package. But critical path packages are perhaps more worth doing this for. Maybe QA can slowly work through the list working with the package owners to try to develop reasonable tests.
Having automated tests or even test plans would be useful for any package. But critical path packages are perhaps more worth doing this for. Maybe QA can slowly work through the list working with the package owners to try to develop reasonable tests.
Yes, it would be great if critical path packages had some sort of test information, test script or test data. It would be incredibly hard for me to test most of the F12 critical path packages otherwise. While I can run F12 under a VM, I don't actively use it so being able to target the problem area and test it would make things easier. F13 packages are somewhat easier since that's my main working machine although there are critical path packages that do come up that I have no idea how to test. Having some sort of information on how to test a package would definitely get more eyes on a critpath package.
I'm relatively new to proventesters. I know I can see unapproved critpath packages on Bodhi but is there a master list of all critpath packages in Fedora? That would help us in tracking down who to work with in hopefully getting this important test information.
-- John Watzke
On Sun, 10 Oct 2010 22:54:11 -0500 John Watzke wrote:
I'm relatively new to proventesters. I know I can see unapproved critpath packages on Bodhi but is there a master list of all critpath packages in Fedora? That would help us in tracking down who to work with in hopefully getting this important test information.
Do you mean this with "master list"? https://admin.fedoraproject.org/updates/critpath?release=F12&untested=Tr...
Thomas
That's the unapproved critical path updates on Bodhi that I was talking about and it is just a small subset of critical path packages where the package needs testing. What I mean is a listing of all packages that would be considered critical path when an update is pushed out for that package. It could be helpful to find package maintainers using this list but it is a bit late to go asking for test info after the package has been pushed.
I did flip the "untested" variable in URL to False. It does return all critical path packages but it probably puts a little strain on the system so it wouldn't be good for everyone to be using that method. A static listing would be better. I was just hoping someone knew of some such list.
-- John Watzke
On Mon, Oct 11, 2010 at 5:19 AM, Thomas Spura tomspur@fedoraproject.orgwrote:
On Sun, 10 Oct 2010 22:54:11 -0500 John Watzke wrote:
I'm relatively new to proventesters. I know I can see unapproved critpath packages on Bodhi but is there a master list of all critpath packages in Fedora? That would help us in tracking down who to work with in hopefully getting this important test information.
Do you mean this with "master list"? https://admin.fedoraproject.org/updates/critpath?release=F12&untested=Tr...
Thomas
On Mon, 11 Oct 2010 09:30:57 -0500 John Watzke wrote:
That's the unapproved critical path updates on Bodhi that I was talking about and it is just a small subset of critical path packages where the package needs testing. What I mean is a listing of all packages that would be considered critical path when an update is pushed out for that package. It could be helpful to find package maintainers using this list but it is a bit late to go asking for test info after the package has been pushed.
I did flip the "untested" variable in URL to False. It does return all critical path packages but it probably puts a little strain on the system so it wouldn't be good for everyone to be using that method. A static listing would be better. I was just hoping someone knew of some such list.
-- John Watzke
On Mon, Oct 11, 2010 at 5:19 AM, Thomas Spura tomspur@fedoraproject.orgwrote:
On Sun, 10 Oct 2010 22:54:11 -0500 John Watzke wrote:
I'm relatively new to proventesters. I know I can see unapproved critpath packages on Bodhi but is there a master list of all critpath packages in Fedora? That would help us in tracking down who to work with in hopefully getting this important test information.
Do you mean this with "master list"? https://admin.fedoraproject.org/updates/critpath?release=F12&untested=Tr...
Thomas
All critpath packages are generated dayly, see: http://fedoraproject.org/wiki/Critical_path_packages#Where_can_I_find_the_cr...
So you are looking for the links provided there. Once [1] is closed, you'll have static listings...
Thomas
Thanks Thomas. This will work great. Now we just need to figure out how to drive this initiative. Perhaps a wiki page to at least identify the packages and if we've worked with the maintainers yet to get an agreement to try to provide some sort of testing information. From there we'll need some place to store some of this information. Test information for a specific bug can easily be stored in the bug itself since its value decreases somewhat after the bug has been fixed and the release has been done. General test scripts would need to be stored somewhere else. Perhaps on this same wiki page where general test scripts could show what needs to be done and for packages such as Perl/GCC/Glibc small code scriptlets could be provided to allow testers to run them and get some general regression coverage.
-- John Watzke
On Mon, 2010-10-11 at 10:58 -0500, John Watzke wrote:
Thanks Thomas. This will work great. Now we just need to figure out how to drive this initiative.
I think you just did! Best way to start, is by discussing and following up.
Perhaps a wiki page to at least identify the packages and if we've worked with the maintainers yet to get an agreement to try to provide some sort of testing information. From there we'll need some place to store some of this information. Test information for a specific bug can easily be stored in the bug itself since its value decreases somewhat after the bug has been fixed and the release has been done.
My preference would be to store test procedures in a single location. For now, it's the wiki. By using the existing wiki infrastructure, it makes potential migration to another tool much easier.
If steps to test or reproduce an issue are posted to a bug, let's provide a clear path for how and where to contribute those test instructions to the wiki.
General test scripts would need to be stored somewhere else.
As long as the documented test instructions provide links to the scripts, that should be fine (whether they are in bugzilla as attachements, uploaded to the wiki as files, in some upstream test suite, or on a fedorapeople.org page).
Perhaps on this same wiki page where general test scripts could show what needs to be done and for packages such as Perl/GCC/Glibc small code scriptlets could be provided to allow testers to run them and get some general regression coverage.
See my earlier response, let's definitely leverage the wiki to get this started. We have some current wiki structure we could refine and use for this purpose. For example, see the hierarchies linked below ...
* https://fedoraproject.org/wiki/Category:Test_Cases * https://fedoraproject.org/wiki/Category:Test_Plans
Thanks, James
On Mon, Oct 11, 2010 at 11:26 AM, James Laska jlaska@redhat.com wrote:
On Mon, 2010-10-11 at 10:58 -0500, John Watzke wrote:
Thanks Thomas. This will work great. Now we just need to figure out how to drive this initiative.
I think you just did! Best way to start, is by discussing and following up.
Okay, let me look over the test case and test plan templates and see what can be put together. I do like your suggestion of starting with the heavy hitters such as firefox/kernel/glibc. That will get us started and we can work our way down to the more obscure test cases.
Sometime this week, I'll put up a wiki page with a general list of known critpath packages. This will give us a working set to start on. I'm thinking we can use that list to let folks "sign up" to track down the maintainer and work through the processing of defining a test case for the package and perhaps updating the status of preparing the tests. Nothing too fancy... just something to keep some status on.
-- John Watzke
On Mon, 2010-10-11 at 12:56 -0500, John Watzke wrote:
On Mon, Oct 11, 2010 at 11:26 AM, James Laska jlaska@redhat.com wrote: On Mon, 2010-10-11 at 10:58 -0500, John Watzke wrote: > Thanks Thomas. This will work great. Now we just need to figure > out how to drive this initiative.
I think you just did! Best way to start, is by discussing and following up.
Okay, let me look over the test case and test plan templates and see what can be put together. I do like your suggestion of starting with the heavy hitters such as firefox/kernel/glibc. That will get us started and we can work our way down to the more obscure test cases.
Sometime this week, I'll put up a wiki page with a general list of known critpath packages. This will give us a working set to start on. I'm thinking we can use that list to let folks "sign up" to track down the maintainer and work through the processing of defining a test case for the package and perhaps updating the status of preparing the tests. Nothing too fancy... just something to keep some status on.
Awesome, I appreciate your initiative here :) If you find it helpful to manage status and coordinate efforts with multiple proventesters, feel free to file tickets in the fedora-qa TRAC instance. Let me know if you'd like to create any additional TRAC milestones to organize this effort.
https://fedorahosted.org/fedora-qa/
Thanks, James
On Mon, 2010-10-11 at 14:17 -0400, James Laska wrote:
Sometime this week, I'll put up a wiki page with a general list of known critpath packages. This will give us a working set to start on. I'm thinking we can use that list to let folks "sign up" to track down the maintainer and work through the processing of defining a test case for the package and perhaps updating the status of preparing the tests. Nothing too fancy... just something to keep some status on.
Awesome, I appreciate your initiative here :) If you find it helpful to manage status and coordinate efforts with multiple proventesters, feel free to file tickets in the fedora-qa TRAC instance. Let me know if you'd like to create any additional TRAC milestones to organize this effort.
Me too - thanks a lot for looking at this! Just a suggestion, but the two packages we'd identified that could benefit from specific testing procedures so far are PackageKit and kernel.
On 10/12/2010 03:43 PM, Adam Williamson wrote:
On Mon, 2010-10-11 at 14:17 -0400, James Laska wrote:
Sometime this week, I'll put up a wiki page with a general list of
known critpath packages. This will give us a working set to start on. I'm thinking we can use that list to let folks "sign up" to track down the maintainer and work through the processing of defining a test case for the package and perhaps updating the status of preparing the tests. Nothing too fancy... just something to keep some status on.
Awesome, I appreciate your initiative here :) If you find it helpful to manage status and coordinate efforts with multiple proventesters, feel free to file tickets in the fedora-qa TRAC instance. Let me know if you'd like to create any additional TRAC milestones to organize this effort.
Me too - thanks a lot for looking at this! Just a suggestion, but the two packages we'd identified that could benefit from specific testing procedures so far are PackageKit and kernel.
Today while trying to do some F12 testing I am getting:
Warning: Server error: ServerError(https://admin.fedoraproject.org/updates/comment, 504, Gateway Timeout) Warning: Comment not submitted: too many errors
On Sun, 2010-10-10 at 22:09 -0500, Bruno Wolff III wrote:
On Sun, Oct 10, 2010 at 18:49:10 -0400, Timothy Davis cpuobsessed@gmail.com wrote:
When a critical path sits for any more than four weeks I suggest the maintainer include test files and test cases to help any tester. Things such as scripts/programs to test with, or scenarios like do this and check that.
For example for squashfs, the bug was hard to test. Kevin saw it, but I didn't. We weren't sure exactly what changed that started breaking things for the nightly composes that Kevin is doing. I was more interested in making sure there weren't regressions than making sure that the bug was really fixed.
Having automated tests or even test plans would be useful for any package. But critical path packages are perhaps more worth doing this for. Maybe QA can slowly work through the list working with the package owners to try to develop reasonable tests.
This is great suggestion! We've talked about spec'ing out some test documentation for a subset of critpath packages, but just haven't had the cycles yet.
If anyone is interested in getting started, I don't think we need more than a draft wiki page to list some concise test instructions. I can see several ways to go here ... either create individual test cases and grouping them in a category (see similar examples at https://fedoraproject.org/wiki/Category:Test_Cases). Alternatively, it might be less wiki work to start noting tests on a single wiki page with an eye towards turning it into a test plan (ala https://fedoraproject.org/wiki/Category:Test_Plans). My only suggestion here would be that we eventually converge on a common format (whether it's test cases, plans or something new).
Is there a critpath component(s) [1] someone might be interested in starting with? Howabout some heavy hitters that come up frequently, like kernel, glibc, firefox etc...?
Thanks, James
[1] http://kojipkgs.fedoraproject.org/mash/branched-20101008/logs/critpath.txt