Greetings!
Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events.
Please review it:
https://fedoraproject.org/wiki/Rhe/tcms_use_cases
Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps:
https://fedoraproject.org/wiki/Rhe/tcms_Comparison
Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here.
Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase.
Waiting for your feedback!
Cheers, Hurry
[1] https://fedorahosted.org/fedora-qa/ticket/152
On Tue, 2011-01-18 at 19:36 +0800, He Rui wrote:
Greetings!
Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events.
Please review it:
https://fedoraproject.org/wiki/Rhe/tcms_use_cases
Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps:
https://fedoraproject.org/wiki/Rhe/tcms_Comparison
Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here.
Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase.
Waiting for your feedback!
Woah, I like the fancy table layout used for comparing use cases between the wiki and nitrate! Thank you for taking the time to identify our current test workflows and offer a comparison between nitrate and the wiki. You've got a really comprehensive list of use cases that will certainly help us evaluate nitrate.
While looking through the comparison wiki, I've noted a few comments/questions below. Hope this helps!
I'm unclear what the use case 'Creating A Common Page' is, can you elaborate? Is that when we create a category page to group all test results for a milestone (e.g. https://fedoraproject.org/wiki/Category:Fedora_14_Pre-Alpha_Test_Results)?
Under 'creating a common page', the row titled 'Submitting format'. Would that be better named as 'Supports data entry using a form'?
Under 'creating a common page', there is a row titled 'Subpage using'. I don't think I understand that use case, can you explain?
Under section 'Creating A Test Case', there are some red cells for the wiki for (these are also under 'Creating a Test Plan'). What does the 'red' mean. Does it mean that they are not possible, or that the solution provided is not optimal? * Renaming a case - move to another page * Case draft status - manually add draft category/note * Case review status - approved in ticket or somewhere else
Also under section 'Creating A Test Case', the row titled 'Test case re-use (write once, link anywhere)' talks about nitrate support for cloning tests. Does this create a new copy of the original, or does it re-use the existing test? If any changes are made to one, are the propagated to the other clone?
Under 'Creating A Test Result Page Template', the row titled 'Group cases (by media)', I imagine nitrate supports other methods of grouping/organizing tests in a test run? Oh wait ... this is grouping, not sorting. How would this be done with nitrate, would a test run be created for each media type? Or would we sort tests of different media types differently?
Under 'Creating A Test Result Page', the row titled 'Redirecting links' ... is this when a test run has expired, or been replaced by a newer test run? Like if we get RC2, we'll go and adjust RC1 so people know it's no longer active? Not sure what to call this feature, retire a test run?
Under 'Posting Test Results', the row titled 'Opening current links' ... I'm trying to think of a more generic name for that feature. Maybe 'Quick access to active test runs' instead?
Under 'Generating A Test Summary', the row titled 'Result summary/report generation'. Haha, you list that as *RED* ... very accurate! :)
Under 'Administering', the row titled 'License', is this the license for nitrate and mediawiki, or the license for the content in the cases, plans and results?
Under 'Release Validation Test Event', the row titled 'Roadmap for each run' ... can you explain, or link to, what is meant by this?
Thanks again! James
Hi James,
Wow, firstly, many thanks for your careful review of all the items.:)
On Wed, 2011-01-19 at 16:08 -0500, James Laska wrote:
On Tue, 2011-01-18 at 19:36 +0800, He Rui wrote:
Greetings!
Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events.
Please review it:
https://fedoraproject.org/wiki/Rhe/tcms_use_cases
Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps:
https://fedoraproject.org/wiki/Rhe/tcms_Comparison
Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here.
Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase.
Waiting for your feedback!
Woah, I like the fancy table layout used for comparing use cases between the wiki and nitrate! Thank you for taking the time to identify our current test workflows and offer a comparison between nitrate and the wiki. You've got a really comprehensive list of use cases that will certainly help us evaluate nitrate.
While looking through the comparison wiki, I've noted a few comments/questions below. Hope this helps!
I'm unclear what the use case 'Creating A Common Page' is, can you elaborate? Is that when we create a category page to group all test results for a milestone (e.g. https://fedoraproject.org/wiki/Category:Fedora_14_Pre-Alpha_Test_Results)?
No, it means creating a general new page on wiki, like a documentation page etc, and test cases/plans/categories/results pages are particular cases based on creating a normal page like this. It lists what a user may do when creating a new page.
Under 'creating a common page', the row titled 'Submitting format'. Would that be better named as 'Supports data entry using a form'?
Ah great, I've changed it.
Under 'creating a common page', there is a row titled 'Subpage using'. I don't think I understand that use case, can you explain?
Okay, for example on wiki, Test_Days is under the path of QA like: https://fedoraproject.org/wiki/QA/Test_Days
So I tried to compare if nitrate can also organize sub-page under its main page.
Under section 'Creating A Test Case', there are some red cells for the wiki for (these are also under 'Creating a Test Plan'). What does the 'red' mean. Does it mean that they are not possible, or that the solution provided is not optimal?
Oh the first two below are not optimal, the last one is not possible on wiki.
* Renaming a case - move to another page
I think moving to another page is not a good way to rename a case, there will be two link addresses for this case.
* Case draft status - manually add draft category/note
It's better to have a case status called draft, or proposed like nitrate does.
* Case review status - approved in ticket or somewhere else
It's better to have a case status called 'confirmed', or 'need update' or 'disabled' like nitrate. I remember we used to track the cases status in tickets and after approved in tickets or reviewed in test@,
Also under section 'Creating A Test Case', the row titled 'Test case re-use (write once, link anywhere)' talks about nitrate support for cloning tests. Does this create a new copy of the original, or does it re-use the existing test? If any changes are made to one, are the propagated to the other clone?
Under 'Creating A Test Result Page Template', the row titled 'Group cases (by media)', I imagine nitrate supports other methods of grouping/organizing tests in a test run? Oh wait ... this is grouping, not sorting. How would this be done with nitrate, would a test run be created for each media type? Or would we sort tests of different media types differently?
Under 'Creating A Test Result Page', the row titled 'Redirecting links' ... is this when a test run has expired, or been replaced by a newer test run? Like if we get RC2, we'll go and adjust RC1 so people know it's no longer active? Not sure what to call this feature, retire a test run?
Under 'Posting Test Results', the row titled 'Opening current links' ... I'm trying to think of a more generic name for that feature. Maybe 'Quick access to active test runs' instead?
Under 'Generating A Test Summary', the row titled 'Result summary/report generation'. Haha, you list that as *RED* ... very accurate! :)
Under 'Administering', the row titled 'License', is this the license for nitrate and mediawiki, or the license for the content in the cases, plans and results?
Under 'Release Validation Test Event', the row titled 'Roadmap for each run' ... can you explain, or link to, what is meant by this?
Thanks again! James -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Unfortunately while I am familiar with a few test case management systems, I have not been involved with the Fedora project long enough to know its workflows. And a quick search online is not turning up much about Nitrate, with its changelog[1] implying it was first made open source software in July 2010.
I might be able to provide some general comments though if I knew more about the product. So for those of us unfamiliar with the history of Nitrate, could you please answer the following:
1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it?
2. Is a sample play/sandbox test instance with more-or-less full access available online?
3. How does Nitrate compare to other open & closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its straights & weaknesses?
4. Is test case & test plan import and export (to XML, etc.) support available? If so is this compatible with any other TCMS's import/export system?
5. Are nested test plan and/or test case hierarchies supported?
6. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans?
7. How long will historical test case results be made available?
8. Is there any plan to tie this into Bodhi and other tools to detect updated packages that may imply test cases need re-running and/or updating?
9. Is this going to be available as a Fedorahosted service like Trac is? If so will all the instances be able to share test cases?
10. Is there any concern that changing test tracking systems may encourage/discourage existing testers to participate?
Some of this information may be useful to post on the Trac main page and/or in the Fedora wiki.
Thanks in advance for your time.
--- SJG
[1] https://fedorahosted.org/nitrate/browser/trunk/nitrate/docs/ChangeLog
On Tue, Jan 18, 2011 at 6:36 AM, He Rui rhe@redhat.com wrote:
Greetings!
Wiki workflow/use cases are reorganized and grouped by general and main test events(runs) use cases. The general cases cover the basic uses of wiki, and the events cases covers the detailed certain steps for organizing the events.
Please review it:
https://fedoraproject.org/wiki/Rhe/tcms_use_cases
Meanwhile, Wiki and Nitrate comparison is also listed and grouped by above use cases. I've listed as many features as I can think out for comparison. Please have a look and see if any features are missed. You can review it by the use case you're familiar with if reviewing all of them is tough, but be aware that features compared in former use cases are not listed in later cases again to avoid overlaps:
https://fedoraproject.org/wiki/Rhe/tcms_Comparison
Ticket#152[1] is currently set up for tracking this event. Feel free to add comments or discuss it here.
Note: the feedback deadline is Jan 21, the end of this week. After that date, I'll move forward to the next phase.
Waiting for your feedback!
Cheers, Hurry
[1] https://fedorahosted.org/fedora-qa/ticket/152
-- Contacts
Hurry FAS Name: Rhe Timezone: UTC+8 TEL: 86-010-62608141 IRC nick: rhe #fedora-qa #fedora-zh
test-announce mailing list test-announce@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/test-announce
On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
Unfortunately while I am familiar with a few test case management systems, I have not been involved with the Fedora project long enough to know its workflows. And a quick search online is not turning up much about Nitrate, with its changelog[1] implying it was first made open source software in July 2010.
I might be able to provide some general comments though if I knew more about the product. So for those of us unfamiliar with the history of Nitrate, could you please answer the following:
Really good list of questions Samuel, thanks for jumping in. I know Hurry can provide feedback on the questions. However, I'll add my thoughts as well.
1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it?
Hurry can touch on the goals for using nitrate. As for history, I can add my experiences ...
This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability.
https://fedoraproject.org/wiki/QA/Testopia_Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013
After this, several folks got together and decided they would implement a new front-end on top of the testopia db schema. This would resolve the original license incompatibility and address usability issues that were raised with the testopia UI. The project started development internally, and was open-sourced in 2010.
1. Is a sample play/sandbox test instance with more-or-less full access available online?
Not yet, I believe that's in plan for sometime during Fedora 15. Hurry and I were discussing the requirements for such an instance earlier this week. If you're interested in helping here, just let us know :)
1. How does Nitrate compare to other open & closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its straights & weaknesses?
See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results.
[1] https://fedorahosted.org/mailman/listinfo/nitrate-devel
1. Is test case & test plan import and export (to XML, etc.) support available? If so is this compatible with any other TCMS's import/export system?
I believe import/export is supported using the testopia.dtd format.
http://git.fedorahosted.org/git/?p=nitrate.git;a=blob;f=trunk/nitrate/docs/C... - Fixed #564258 - [REF] Ability to export/print specified cases - Fixed #612803 - add an export feature for test case runs, can export …
1. Are nested test plan and/or test case hierarchies supported?
I don't recall. Perhaps Hurry or the nitrate developers know?
I know this feature has been discussed a *lot* with nitrate, and other solutions we've used in the past. I don't think support for nested test plans is something we've had a tremendous need for now, so I don't anticipate this being a MUSTHAVE feature in the near term.
1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans?
This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance.
With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases.
Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term.
1. How long will historical test case results be made available?
I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for.
1. Is there any plan to tie this into Bodhi and other tools to detect updated packages that may imply test cases need re-running and/or updating?
That is certainly possible, but there are currently no detailed plans to integrate bodhi/f-e-k with nitrate specifically. Nitrate offers a robust XMLRPC interface to allow other tools, such as bodhi, to query for applicable tests/results/runs etc... So I wouldn't think it would be a difficult task.
That said, Adam Williamson is exploring this linkage using our current wiki-based test management system. While the two implementations will be completely different, the end result is the same ... improved integration between our update tools (bodhi/f-e-k) and our test infrastructure.
1. Is this going to be available as a Fedorahosted service like Trac is? If so will all the instances be able to share test cases?
Hmm, I don't envision multiple hosted instances of nitrate (or any other test case management) in Fedora. I expect we would have a single test management system available for all to use, much like we have a build system, updates and bug reporting.
1. Is there any concern that changing test tracking systems may encourage/discourage existing testers to participate?
Yes! That's part of why Hurry is investing a lot of time researching our current workflows and comparing features between our current wiki-based implementation.
Some of this information may be useful to post on the Trac main page and/or in the Fedora wiki.
Thanks in advance for your time.
Thank you for your input, good comments/questions!
Thanks, James
A few more inline comments on a subset of the questions, and two more thoughts:
On Thu, Jan 20, 2011 at 11:54 AM, James Laska jlaska@redhat.com wrote:
On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it?
This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability.
https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013
From the bug I'm guessing this is because Testopia used Ext-JS, and
Ext-JS kept changing licenses. Hopefully there will only be licensing restrictions for Nitrate on the software itself, and not the created work of what one does while actively using it.
1. How does Nitrate compare to other open & closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its strengths & weaknesses?
See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results.
Are the developers actively monitoring/using this list? I looked at it before, and all I saw were three test posts from July in the archives.
1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans?
This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance.
With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases.
Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term.
The reason I bring this up is because OLPC's Spin releases tend to lag behind the official Fedora release. For instance, we just released our hopefully final Fedora 11-based release this past month, and are in the early stages of the Fedora 14-based one. At least some Fedora ARM development work may still be going on with Fedora 13 as well.
While decently written test cases will survive somewhat over time, major changes in GUI look & feel or other areas can break their backwards compatibility.
Ideally Nitrate will default to using the current version of a testcase when making a new plan, preferably following the updates of said testcase until a result is committed which forces the test case version to be needed.
1. How long will historical test case results be made available?
I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for.
Again; this would be to help lagging Spins and similar. Right now it looks like Bodhi at least publicly hides information for Fedora versions which have reached end-of-life. (Either that, or I don't know how to find it.)
I presume that Nitrate has already been determined to be scalable; otherwise it is a big risk to incorporate it into Fedora.
Also: It might be useful to add an "unclear" testcase result, similar to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote:
A few more inline comments on a subset of the questions, and two more thoughts:
On Thu, Jan 20, 2011 at 11:54 AM, James Laska jlaska@redhat.com wrote:
On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it?
This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability.
https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013
From the bug I'm guessing this is because Testopia used Ext-JS, and
Ext-JS kept changing licenses. Hopefully there will only be licensing restrictions for Nitrate on the software itself, and not the created work of what one does while actively using it.
That was a question I had opened with Hurry. I have no idea how we'd license the content in the system. If anything, I'd hope/expect that to match that of our current wiki.
1. How does Nitrate compare to other open & closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its strengths & weaknesses?
See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results.
Are the developers actively monitoring/using this list? I looked at it before, and all I saw were three test posts from July in the archives.
I believe they do, but don't yet have a strong upstream presence since there hasn't been a lot of code/progress to share until recently. This will be something I'm sure they'll want to improve as community interest increases.
1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans?
This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance.
With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases.
Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term.
The reason I bring this up is because OLPC's Spin releases tend to lag behind the official Fedora release. For instance, we just released our hopefully final Fedora 11-based release this past month, and are in the early stages of the Fedora 14-based one. At least some Fedora ARM development work may still be going on with Fedora 13 as well.
While decently written test cases will survive somewhat over time, major changes in GUI look & feel or other areas can break their backwards compatibility.
Ideally Nitrate will default to using the current version of a testcase when making a new plan, preferably following the updates of said testcase until a result is committed which forces the test case version to be needed.
I hope so, yes! :) I'm fairly certain support exists for linking against versioned cases, that is, I know it was present in testopia. Hopefully Hurry, or the nitrate folks can help here. Sounds like we need to add this to our list of requirements.
1. How long will historical test case results be made available?
I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for.
Again; this would be to help lagging Spins and similar. Right now it looks like Bodhi at least publicly hides information for Fedora versions which have reached end-of-life. (Either that, or I don't know how to find it.)
That or garbage collected.
Sounds like we'll need to do some further research on this front. I personally see no reason to purge results for EOL'd releases. I've often referenced old test results to understand when the last time something was tested. Test results aren't disk-hogs like builds (*cough* oo.org *cough*) can be, so I don't expect this to be as much of an issue. Nonetheless, you're point is valid.
I presume that Nitrate has already been determined to be scalable; otherwise it is a big risk to incorporate it into Fedora.
I'll let the nitrate folks speak to scalability issues. That's definitely something we'll want to explore during a pilot. Remember too, I don't expect we'll throw away the wiki, and switch to nitrate without a good bake-in period to identify bugs/gaps etc...
Also: It might be useful to add an "unclear" testcase result, similar to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus.
Thanks, James
Greetings,
Many thanks for Samuel's comments, I found out many valuable things to think, identify and add to use cases/feature comparison.
The questions also remind me that many testers so far may be still not familiar with Nitrate, therefore can't give any suggestion upon it. So I'll introduce Nitrate a bit here.
Nitrate is a new test case management system which is open-sourced recently. It has advantages on managing, tracking and querying cases,results and is similar to testopia. The introduction pages along with some screenshots can be found at:
https://fedorahosted.org/nitrate/ https://fedoraproject.org/wiki/Nitrate
One can set up a local nitrate system with the steps in the link, we are also planing to build a testing instance during F-15.
On Thu, 2011-01-20 at 15:02 -0500, James Laska wrote:
On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote:
A few more inline comments on a subset of the questions, and two more thoughts:
On Thu, Jan 20, 2011 at 11:54 AM, James Laska jlaska@redhat.com wrote:
On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote:
1. What is the history of Nitrate and the Fedora Project? What does the Fedora project expect to gain from using it?
This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability.
https://fedoraproject.org/wiki/QA/Testopia+AF8-Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013
From the bug I'm guessing this is because Testopia used Ext-JS, and
Ext-JS kept changing licenses. Hopefully there will only be licensing restrictions for Nitrate on the software itself, and not the created work of what one does while actively using it.
That was a question I had opened with Hurry. I have no idea how we'd license the content in the system. If anything, I'd hope/expect that to match that of our current wiki.
Oh, licensing and license issues. I've listed both of them in feature comparison as requirements. For test days and release validation test events use cases, I identified the steps with different permissions by QA, event host and tester(including anonymous user)[#], and hope this can help to license these instances in Nitrate. Of course, I need to extend more to cover other instances.
[#] https://fedoraproject.org/wiki/Rhe/tcms_use_cases#Test_Events_Use_Cases
1. How does Nitrate compare to other open & closed sourced TCMS solutions? Why was it written as opposed to using an existing solution, and what are its strengths & weaknesses?
See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results.
Are the developers actively monitoring/using this list? I looked at it before, and all I saw were three test posts from July in the archives.
I believe they do, but don't yet have a strong upstream presence since there hasn't been a lot of code/progress to share until recently. This will be something I'm sure they'll want to improve as community interest increases.
Add it into requirements. But I can confirm that all the nitrate developers are monitoring the list now. When community interest increases, it will become active. :)
1. Can multiple projects share test cases, and even reference older versions of test cases if they are lagging behind the current rawhide/Fedora release? Will Spins be able to make their own (simultaneously running) test plans?
This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance.
With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases.
Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term.
The reason I bring this up is because OLPC's Spin releases tend to lag behind the official Fedora release. For instance, we just released our hopefully final Fedora 11-based release this past month, and are in the early stages of the Fedora 14-based one. At least some Fedora ARM development work may still be going on with Fedora 13 as well.
While decently written test cases will survive somewhat over time, major changes in GUI look & feel or other areas can break their backwards compatibility.
Ideally Nitrate will default to using the current version of a testcase when making a new plan, preferably following the updates of said testcase until a result is committed which forces the test case version to be needed.
I hope so, yes! :) I'm fairly certain support exists for linking against versioned cases, that is, I know it was present in testopia. Hopefully Hurry, or the nitrate folks can help here. Sounds like we need to add this to our list of requirements.
In Nitrate, the text part of a test case is versioned, and one can view different text versions of the case but can't reference old versions to test run. testopia can, but nitrate abandoned it. I've listed it in the requirements for valuating.
1. How long will historical test case results be made available?
I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for.
Again; this would be to help lagging Spins and similar. Right now it looks like Bodhi at least publicly hides information for Fedora versions which have reached end-of-life. (Either that, or I don't know how to find it.)
That or garbage collected.
Sounds like we'll need to do some further research on this front. I personally see no reason to purge results for EOL'd releases. I've often referenced old test results to understand when the last time something was tested. Test results aren't disk-hogs like builds (*cough* oo.org *cough*) can be, so I don't expect this to be as much of an issue. Nonetheless, you're point is valid.
Add it to the requirement list.
I presume that Nitrate has already been determined to be scalable; otherwise it is a big risk to incorporate it into Fedora.
I'll let the nitrate folks speak to scalability issues. That's definitely something we'll want to explore during a pilot. Remember too, I don't expect we'll throw away the wiki, and switch to nitrate without a good bake-in period to identify bugs/gaps etc...
Also: It might be useful to add an "unclear" testcase result, similar to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus.
Agree with James.
Thanks, James -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
In addition, I've also added 'Integration between our update tools (bodhi/f-e-k) and our test infrastructure' to requirements for evaluation.
Feel free to tell if there's any concern.
Thanks, Hurry
On Fri, Jan 21, 2011 at 11:40 AM, He Rui rhe@redhat.com wrote:
Also: It might be useful to add an "unclear" testcase result, similar to how Mozilla's Litmus system does it (https://litmus.mozilla.org).
I do like litmus! It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus.
Agree with James.
What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to "Passed", "Failed", and any other common states (Blocked, In Progress, etc.) you have an "Unclear" result state.
This forces the result (including a comment section I presume is forced for all non-passing items) to be evaluated during the release cycle to see if it was an actual failure, or if the test case needs to be updated and/or clarified.
See the "recently unclear" and "testcases most frequently marked as unclear" links at the bottom of pretty much every page of litmus.mozilla.org for examples. On the former, you can click on the dialog icon beneath each result ID to see the comment without switching pages.
Personally I like this approach because management of many projects (in my experience) rarely budget time for test cases to be updated. They prefer that you start testing the next release instead. Projects I have been on have required me to use test cases which were last notably updated a few years ago if not 5-10+ years ago by simply acting as if the inaccurate information is not there. By treating potentially incorrect test cases as a form of failure it forces them to be ideally updated and rerun promptly.
The downside is if a tester is unfamiliar with an area or needs hand-holding, they might mark a lot of results unclear which actually are rather straightforward. Or alternatively, no one looks at them, and you have a muddy result at the end of the release. These would put additional effort on the core QA team to remedy the situation.
On Fri, 2011-01-21 at 19:02 +0000, Samuel Greenfeld wrote:
> I do like litmus! It's a nice evolution from testopia for upstream > mozilla. We don't currently have an 'unclear' test result. I'm not > opposed to it, but would need better understand how that field is used, > and the process around it, in litmus. Agree with James.
What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to "Passed", "Failed", and any other common states (Blocked, In Progress, etc.) you have an "Unclear" result state.
Hurry is somewhat wrong to say we don't currently have an 'unclear' result; we do have the 'warn' result, which is in some ways similar. We usually use it to indicate when a test turns up some kind of anomalous behaviour which isn't exactly a failure.
On Mon, 2011-01-24 at 23:05 -0800, Adam Williamson wrote:
On Fri, 2011-01-21 at 19:02 +0000, Samuel Greenfeld wrote:
> I do like litmus! It's a nice evolution from testopia for upstream > mozilla. We don't currently have an 'unclear' test result. I'm not > opposed to it, but would need better understand how that field is used, > and the process around it, in litmus. Agree with James.
What I believe Mozilla is doing (since I have not had a chance to work with their QA team yet) is flagging test cases with a form of soft failure in that the result of a testcase neither clearly passed, nor clearly failed. So in addition to "Passed", "Failed", and any other common states (Blocked, In Progress, etc.) you have an "Unclear" result state.
I didn't receive the mail replied by Samuel. Weird.
Hurry is somewhat wrong to say we don't currently have an 'unclear' result; we do have the 'warn' result, which is in some ways similar. We usually use it to indicate when a test turns up some kind of anomalous behaviour which isn't exactly a failure.
It depends on what 'unclear' means here and how it reflects the results. If you mean a soft failure or an issue that doesn't block the case run, the 'warn' result is similar to it, then calling it 'unclear' is confusing and not accurate in my opinion.
In nitrate system, it has 'blocked', 'failed' and 'error' result status to reflect a problem. User guide suggests 'error' is used for test environment that has problems that prevent Test Case execution. I think we can modify 'error' status to include all soft failures.
I've added this to the requirements for further evaluation.
Thanks, Hurry