Excerpts from Nick Coghlan's message of 2013-10-08 14:48:34 +1000:
1. Gerrit 2.7 [1] includes the ability to display image files inline
in
the code review (showing the old and new versions adjacent to each other
rather than a diff)
2. Facebook's Huxley [2] is based on automatically taking screenshots
based on a webdriver test and committing them to Git
It seems to me we may be able to kill two birds with one stone here: by
creating Huxley tests for various parts of the UI, we'll get screenshot
changes to review as part of the commits, *and* potentially be able to
automate regression testing for cosmetic problems that are painful to
test with explicit WebDriver API calls.
This sounds like a promising idea. We may not even need to use Huxley,
since we already have a huge suite of WebDriver tests. It might be
enough for us to write a utility function to grab a screenshot and
compare it to an expected one, then we could insert that at appropriate
places in our tests. If we establish a good convention for how to name
the screenshot files, and make it easy to take new ones or update them,
it should be fairly painless to maintain I think.
My only concern is that platform differences could cause the browser
screenshots to come out differently depending on where the tests are
run. For example differences in installed fonts or fontconfig settings
or Freetype versions might all cause slight deviations in text rendering
which would cause false failures for any assertion that requires the
screenshot to be pixel-perfect to the expected one.
The same article also referenced this UI acceptance testing tool
which
looks potentially useful for acceptance testing purposes:
https://github.com/bslatkin/dpxdt
Unlike Huxley, which is designed to integrate with the normal code
review and automated regression testing processes for a project, dpxdt
(pronounced "depicted") is designed to run as a separate service,
capturing screenshots and generating perceptual diffs which are then
approved (or not) based on whether they represent deliberate or neutral
changes or UI regressions.
I'm not sure that an external service makes sense for our development
workflow, but the idea of a "perceptual diff" sounds interesting and
might be worth exploring.
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.