I've been working to revamp the AutoQA infrastructure so that we're not running completely on F17 clients and after building several new client machines in staging, I was curious about how well the tests were running on the new client machines.
I've included the numbers below but 2 things jump out at me: - #438 is causing problems for repoclosure - there is a problem with depcheck on f19. the only runs that didn't crash are the ones without any builds in them. The crash looks consistent but I don't know what the root cause is.
Note that these numbers reflect execution in staging autotest since I rebuilt the clients (including the queued jobs that weren't run while rebuilding), not pass/fail results and not from production. I was only interested in numbers about execution of the tests (GOOD - executed cleanly, reported results. ERROR - problems in execution, probably didn't report results)
Tim
Global Stats ============ 2077 Jobs - 535 fc17 - 760 fc18 - 781 fc19
Crashes ------- 227 Crashes - 0 fc17 - 80 fc18 - 147 fc19
Passes ------- 1849 Passes - 535 fc17 - 680 fc18 - 634 fc19
Test Specific Stats ===================
depcheck 363 runs ( 167 GOOD, 196 ERROR ) - 84 fc17 runs ( 84 GOOD, 0 ERROR ) - 151 fc18 runs ( 75 GOOD, 76 ERROR ) - 128 fc19 runs ( 8 GOOD, 120 ERROR )
repoclosure 48 runs ( 27 GOOD, 21 ERROR ) - 7 fc17 runs ( 7 GOOD, 0 ERROR ) - 20 fc18 runs ( 20 GOOD, 0 ERROR ) - 21 fc19 runs ( 0 GOOD, 21 ERROR )
rpmguard 795 runs ( 790 GOOD, 5 ERROR ) - 202 fc17 runs ( 202 GOOD, 0 ERROR ) - 270 fc18 runs ( 268 GOOD, 2 ERROR ) - 323 fc19 runs ( 320 GOOD, 3 ERROR )
rpmlint 795 runs ( 790 GOOD, 5 ERROR ) - 220 fc17 runs ( 220 GOOD, 0 ERROR ) - 301 fc18 runs ( 299 GOOD, 2 ERROR ) - 274 fc19 runs ( 271 GOOD, 3 ERROR )
upgradepath 75 runs ( 75 GOOD, 0 ERROR ) - 22 fc17 runs ( 22 GOOD, 0 ERROR ) - 18 fc18 runs ( 18 GOOD, 0 ERROR ) - 35 fc19 runs ( 35 GOOD, 0 ERROR )
depcheck 363 runs ( 167 GOOD, 196 ERROR ) - 84 fc17 runs ( 84 GOOD, 0 ERROR ) - 151 fc18 runs ( 75 GOOD, 76 ERROR ) - 128 fc19 runs ( 8 GOOD, 120 ERROR )
repoclosure 48 runs ( 27 GOOD, 21 ERROR ) - 7 fc17 runs ( 7 GOOD, 0 ERROR ) - 20 fc18 runs ( 20 GOOD, 0 ERROR ) - 21 fc19 runs ( 0 GOOD, 21 ERROR )
Thanks, Tim, for collecting these numbers. It seems we should avoid F19 test clients for the moment. As for the repoclosure bug, I already responded to its ticket (i.e. not worth the invested time, I think). Depcheck problem... ugh. Anyone wants to consult this with yum developers? Last time I talked to them, there were surprised how hackish the whole concept was ("faking all packages to be installed at the same time, even conflicting ones? really? that can't work"). Or is this the time to write simple_depcheck? :-)
On Mon, 22 Jul 2013 06:40:44 -0400 (EDT) Kamil Paral kparal@redhat.com wrote:
depcheck 363 runs ( 167 GOOD, 196 ERROR ) - 84 fc17 runs ( 84 GOOD, 0 ERROR ) - 151 fc18 runs ( 75 GOOD, 76 ERROR ) - 128 fc19 runs ( 8 GOOD, 120 ERROR )
repoclosure 48 runs ( 27 GOOD, 21 ERROR ) - 7 fc17 runs ( 7 GOOD, 0 ERROR ) - 20 fc18 runs ( 20 GOOD, 0 ERROR ) - 21 fc19 runs ( 0 GOOD, 21 ERROR )
Thanks, Tim, for collecting these numbers. It seems we should avoid F19 test clients for the moment.
That's a good point, with how easy it is to create clients right now, we could rebuild with just F18 clients until some of these issues are fixed.
As for the repoclosure bug, I already responded to its ticket (i.e. not worth the invested time, I think).
Depcheck problem... ugh. Anyone wants to consult this with yum developers? Last time I talked to them, there were surprised how hackish the whole concept was ("faking all packages to be installed at the same time, even conflicting ones? really? that can't work"). Or is this the time to write simple_depcheck? :-)
It feels like it could be rather simple, but I have about 10% confidence that it actually would be simple - it's been a while since I've dug into that code and who knows how long it would actually take.
j_dulaney/handsome_pirate has been talking about a new depcheck, but he hasn't been able to upload code for it yet. Either way, I'm all for killing off depcheck as is :)
Tim
On Fri, 19 Jul 2013 08:56:48 -0600 Tim Flink tflink@redhat.com wrote:
Note that these numbers reflect execution in staging autotest since I rebuilt the clients (including the queued jobs that weren't run while rebuilding), not pass/fail results and not from production. I was only interested in numbers about execution of the tests (GOOD - executed cleanly, reported results. ERROR - problems in execution, probably didn't report results)
I've switched the clients in stg to all fc19 clients to highlight the problems with more data. I recalculated the stats from roughly the last 24 hours
http://tflink.fedorapeople.org/autoqa/jobstats/20130724_jobstats-stg_summary...
The two that are still jumping out at me are:
depcheck 163 runs ( 45 GOOD, 116 ERROR ) - 43 fc18 runs ( 43 GOOD, 0 ERROR ) - 118 fc19 runs ( 2 GOOD, 116 ERROR )
I think that almost all of the fc19 failures are due to #442 which we should probably at least look at to see if it's going to be feasibly fixable. If all else fails, we can either get rid of the fc19 clients or keep depcheck from being scheduled on fc19.
conflicts 154 runs ( 34 GOOD, 116 ERROR ) - 30 fc18 runs ( 0 GOOD, 30 ERROR ) - 120 fc19 runs ( 34 GOOD, 86 ERROR )
I'm not really sure what's going on with conflicts but it is crashing a lot for both fc18 and fc18. I've filed #441 to track the issue here.
I've not looked into either of these bugs yet, so please take them if you're interested - just make a comment in the bug to say that you're working on it so that we don't have multiple people working on the same thing :)
Tim
qa-devel@lists.fedoraproject.org