Hey, folks. Today I got a PoC of testcase-stats on top of wikitcms up and running. It's not complete yet as I haven't written the bits to provide the CLI and page selection stuff, but all the hard stuff is there. I'm attaching it. To test it, just stick it in the stats/testcase-stats directory of a fedora-qa git checkout, symlink an up-to-date checkout of the wikitcms 'results-types' branch into the same directory, and run it. It'll output to /tmp/tcstats_test , but you can easily change that if you like. You can also change the hard-coded list of pages it runs on.
Tomorrow I'll aim to clean it all up and make it a relval sub-command, then maybe release wikitcms 2.0. it may get interesting when I look at feeding in various possible groups of pages, but I'll burn that bridge when I come to it.
The contortions this goes through to wrangle the data into the correct forms seem slightly inelegant, but I don't see any easy significant wins in simplifying them - thoughts welcome. What it does is roughly this:
* Get a list of ResultRow objects for each page in the page set
* make a dict of lists of tuples (!). the keys are tuples for all the 'unique tests' for which at least one ResultRow exists, where a 'unique test' is the combination of testcase + 'test name' - when you have something like [[QA:Testcase_default_boot_install|Workstation live]], QA:Testcase_default_boot_install is the testcase and "Workstation live" is the 'test name'. the value tuples are the ResultRows for that 'unique test'; field 0 is a string identifying the compose from which the ResultRow comes (e.g. 'BetaTC1'), field 1 is the ResultRow object itself. we need to keep the rows for each compose for each 'unique test' separate.
* make another dict, using the same keys from the last one, with the value for each key being a Test() object instantiated from the list of tuples from the first test case. That class can now produce all the information the HTML rendering functions need to do their stuff.
if anyone sees a way to simplify that while still making it possible for the renderers to get all the data they need fairly simply, please do enlighten me =) one possible thing we could do is let wikitcms provide milestone / compose properties for resultrow objects, that might help. I'll have to look at it tomorrow.
If I was writing this from scratch maybe I'd go about it slightly differently, but I don't think the ugliness is sufficient to merit burning down all josef's work and starting over. it works fine.
the actual rendering functions are more or less intact from josef's version, I just adjusted them to consume the data from the Test() objects. the Test() classes' __init__() is pretty much ripped from josef's version too.
On Sat, 2014-10-11 at 02:37 -0700, Adam Williamson wrote:
Hey, folks. Today I got a PoC of testcase-stats on top of wikitcms up and running. It's not complete yet as I haven't written the bits to provide the CLI and page selection stuff, but all the hard stuff is there. I'm attaching it. To test it, just stick it in the stats/testcase-stats directory of a fedora-qa git checkout, symlink an up-to-date checkout of the wikitcms 'results-types' branch into the same directory, and run it. It'll output to /tmp/tcstats_test , but you can easily change that if you like. You can also change the hard-coded list of pages it runs on.
Tomorrow I'll aim to clean it all up and make it a relval sub-command, then maybe release wikitcms 2.0. it may get interesting when I look at feeding in various possible groups of pages, but I'll burn that bridge when I come to it.
So I didn't quite get to the relval sub-command point yet, but I made it a hell of a lot better today. If you want to play with my current state of the art:
https://www.happyassassin.net/temp/tcstats_test/
is the output from a run on Fedora 20. It handles test cases that occur multiple times both with different 'names' and with the same 'name' but *in a different table* (it was rather badly thrown by the Desktop page before I made wikitcms parse tables).
By default it displays the tests in more or less the order they appear on the actual pages, with table separators - changes to the results or tables between composes confuse it a bit, whenever a case shows up that didn't exist in a previous compose it'll likely get stuck at the bottom and may lead to table separator duplication.
It provides rather a lot more detail on the 'details' page: it counts results per 'environment', lists bugs for passes and warns as well as fails, and calculates the 'coverage' - the percentage of environments for which a result was provided - for each page.
https://www.happyassassin.net/temp/tcstats_test/Desktop.html vs http://testdays.qa.fedoraproject.org/testcase_stats/Category_Desktop_validat... makes an interesting comparison (mine looked a lot like Josef's before I taught it to parse tables) - you can see mine's more complex but also trickable, the name of the 'release blocking' table changed after Alpha TC1, but it's rather difficult to somehow catch that and merge the results with those for the renamed table from later composes. The easier fix in this and some similar cases is just to go to the wiki and make the pages consistent =)
Sorting is currently broken - that's more or less next on my list (after somehow fixing "Install" / "Installation", which is confusing the 21 results slightly).
The current state of the code is attached for the curious, it still needs some cleanups here and there. I made write_detail_page() a function nested within the print_results_html() function because it needs to be run for each test kind of in the middle of the summary page generation loop, to make sure the links from the summary page and the page names actually generated by the detail page writer stay consistent. It also works as a top-level function you pass a whole *ton* of variables into, but that seemed uglier. I might instead just make Test class methods to generate the necessary bits and call into those, have to try it later to see if that makes it nicer.
It still needs HEAD of wikitcms' results-types branch, I had to make quite a lot of changes to wikitcms. All the improved result page parsing went in there, of course.
Amongst other things, the page.ComposePage() class grew a couple of slightly interesting new members.
from_wikipage() is a class method which you can feed an mwclient page object (i.e. a wiki page) and it'll try and spit out the ComposePage() object for the same page - newstats first gets all the mwclient page objects for the release according to Release.milestone_pages(), then feeds 'em into this method to get wikitcms page objects it can use.
sortname() is a property which returns a string that you can use to sort ComposePage objects: for any given set of ComposePage objects, sorting with sortname() as the key should give you the pages in the 'correct' order, i.e. Fedora 13 Final RC1, Fedora 14 Alpha TC1, Fedora 14 Alpha TC2, Fedora 14 Alpha RC1, Fedora 14 Beta TC1, Fedora 14 Final RC2, Fedora 16 Alpha TC3... - the order the composes actually happened in. It does this by join()ing the release with heavily-modified versions of the milestone and compose that should always sort correctly...or, well, I mean, as close as seemed reasonable to bother handling (we had some really pretty wacky page names back before Fedora 14). newstats uses this to sort the pages for each test type right before it gets their result rows and turns them into newstats Test() objects. Before I made it an instance property, I had this magnificently hideous hack instead, which I wish to preserve for posterity:
pages.sort(key=lambda x: ' '.join((x.milestone, x.compose.replace('TC', 'AC'))))
which...actually works surprisingly well, since "Alpha", "Beta" and "Final" happen to sort alphabetically in any case. count the parentheses!
qa-devel@lists.fedoraproject.org