On Fri, 2015-10-09 at 21:41 +0200, Kalev Lember wrote:
These failed test IDs are all nice to see, but they don't say
much to
the uninitiated. :)
Yeah, what I'd actually like to do is have links to the tests; the
reason we can't do that right now is that the openQA instance is behind
the RH firewall so only RHers can see it. We're trying to fix that - in
fact that's what I'd be doing today if I didn't have to retake a few
dozen screenshots instead :P
These tests are comparing screenshots pixel by
pixel, right? Are the screenshots and other test results publicly
available somewhere?
See above. And soooooorrt of, but not really. openQA matches on areas
within screenshots; an openQA 'needle' is a screenshot with an
accompany JSON file that identifies areas within it (and contains tags
used for actually doing matches). A needle 'matches' if all the marked
areas have matches within the screenshot being compared, after a bit of
image processing is applied (color reduction and sharpening), within a
degree of tolerance (~96% similarity according to some algorithm or
other).
How are the screenshots rebased? Is it all manual work?
More or less. the openQA web UI has an interactive needle editor which
lets you basically run a test and pause after each failed match, then
you can fiddle with the needle and re-run the match. I haven't used it
much and never for doing a big screenshot rebase, so I can't say
whether it's more or less convenient than the way I did the last one,
which was to reproduce the tests by hand in a VM, take the screenshots
from virt-manager, copy the json file and make any necessary
adjustments to the match area with the 'crop.py' needle editor (which
is an alternative tool upstream provides in one of their repos).
Is there any way to link to the failed screenshots from the status
mail
so that I could just follow links and see what's changed?
Once we get openQA deployed publicly, yeah. As I said, we're working on
that; https://phab.qadevel.cloud.fedoraproject.org/T623 for the gritty
details.
Sorry for bombing you with all the questions. I think it's
totally
awesome that we have nightly testing, thanks for getting this set up!
No probs!
Ultimately this is an occupational hazard of how openQA works, it's
just that I was surprised that a font would suddenly change its
appearance quite noticeably (at least to a tool like openQA). But now I
got some references from you and Matthias I understand that it was to
fix clear rendering errors, which is pretty reasonable, I just hope
there isn't too much churn in future releases :)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net