On Fri, 2015-10-09 at 21:41 +0200, Kalev Lember wrote:
These failed test IDs are all nice to see, but they don't say much to the uninitiated. :)
Yeah, what I'd actually like to do is have links to the tests; the reason we can't do that right now is that the openQA instance is behind the RH firewall so only RHers can see it. We're trying to fix that - in fact that's what I'd be doing today if I didn't have to retake a few dozen screenshots instead :P
These tests are comparing screenshots pixel by pixel, right? Are the screenshots and other test results publicly available somewhere?
See above. And soooooorrt of, but not really. openQA matches on areas within screenshots; an openQA 'needle' is a screenshot with an accompany JSON file that identifies areas within it (and contains tags used for actually doing matches). A needle 'matches' if all the marked areas have matches within the screenshot being compared, after a bit of image processing is applied (color reduction and sharpening), within a degree of tolerance (~96% similarity according to some algorithm or other).
How are the screenshots rebased? Is it all manual work?
More or less. the openQA web UI has an interactive needle editor which lets you basically run a test and pause after each failed match, then you can fiddle with the needle and re-run the match. I haven't used it much and never for doing a big screenshot rebase, so I can't say whether it's more or less convenient than the way I did the last one, which was to reproduce the tests by hand in a VM, take the screenshots from virt-manager, copy the json file and make any necessary adjustments to the match area with the 'crop.py' needle editor (which is an alternative tool upstream provides in one of their repos).
Is there any way to link to the failed screenshots from the status mail so that I could just follow links and see what's changed?
Once we get openQA deployed publicly, yeah. As I said, we're working on that; https://phab.qadevel.cloud.fedoraproject.org/T623%C2%A0for the gritty details.
Sorry for bombing you with all the questions. I think it's totally awesome that we have nightly testing, thanks for getting this set up!
No probs!
Ultimately this is an occupational hazard of how openQA works, it's just that I was surprised that a font would suddenly change its appearance quite noticeably (at least to a tool like openQA). But now I got some references from you and Matthias I understand that it was to fix clear rendering errors, which is pretty reasonable, I just hope there isn't too much churn in future releases :) -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net