Adam Williamson wrote:
On Sat, 2011-10-15 at 10:03 +0200, Kevin Kofler wrote:
> Adam Williamson wrote:
> > The other problem with screenshot validation is that it's inherently
> > unsuited to some tests that are important to Fedora and that we have
> > more-or-less working *right now* in AutoQA - things like depcheck and
> > repoclosure - because all it can really tell you is 'does this results
> > screen look like a known pass case'
>
> Well, the test could:
> 1. bring up a known terminal emulator,
> 2. type this into it:
> su -c "yum -y install packagename >/dev/null 2>/dev/null; echo $?"
> 3. have a PASS screenshot where the terminal emulator displays 0 and a
> FAIL screenshot where the terminal emulator displays 1.
>
> (But of course that takes a lot more work to set up than just checking
> the dependencies directly.)
This is pretty much what it does - you get the screenshots as output
from openQA (and a video, which is kinda awesome).
But it still doesn't know what the text *is*: it can't store it in a
database or email it to you. You have to actually look at the picture.
You can't search it or compare it or anything. It's just a picture of
some words.
(If someone suggests putting an OCR in the loop I'm going to get my
gun. :>)
The point of my test is that it'd get only 2 possible answers, 1 and 0. (If
there are other exit codes, append an || false and it'll be only 1 and 0.)
So you take a screenshot of the console output and it will be one of 2
canned screenshots, where the one with 0 will be marked as known pass and
the one with 1 as known fail. But of course, if the screenshot only shows
"1", you have no idea what the error actually was.
Kevin Kofler