On Sat, Sep 02, 2017 at 01:32:11PM -0700, Adam Williamson wrote:
On Fri, 2017-09-01 at 22:09 -0400, Jeff Backus wrote:
>
> Thanks for so much information. Very helpful. The automated tests,
> are they easy to automate on actual hardware? or do key parts of the
> automated tests sit outside of the VM?
It's possible, but not easy. The test runner sits outside the system
under test and does several things - mainly, inject keypresses and
mouse movements, and watch the screen. It also needs to monitor the
serial output. There are some other things (like taking snapshots, and
monitoring audio) but those are the main ones. Upstream openQA (which
is basically SUSE) *does* have backends (the test runner's a pretty
modular design) which use real hardware, rather than virtual machines,
but it requires several special bits of hardware to achieve all the
above, and I've never tried it out at all, only looked at the code and
thought 'huh, that looks interesting' - I don't even know whether the
code is really used in anger at all at SUSE. Looking into this is
possible, but it would not be trivial - it'd be a multi-week project
for someone.
Interesting! Yes, I think that is a much larger effort than we have resources for. Thanks
for the clarification.
> It sounds to me like the best ways the SIG can help QA is:
> * Be a primary resource (hardware and warm bodies) for debugging
> i686-related boot, kernel, driver, and instruction-set related issues
> * Help with testing
> * Help triage all other i686-related issues in Bugzilla
> * more testing!
This sounds about right, except that I'd conceive of it exactly the
other way around: *we* (QA) are helping *you* (the SIG), by providing
test cases, release validation events and so on. We're providing
resources to help you ensure the x86 packages and deliverables work
well. That's kinda the model I like to use for thinking about things in
these days where we have zillions of arches and different bits of the
distro that people are interested in; it's never going to be the case
that QA per se is going to test them all, so especially outside of the
release-critical stuff, we like to think of ourselves as providing the
support for SIGs, WGs and interested users to test the bits they care
about.
Gah. Yes, you're right. That point of view makes a lot more sense in the grand scheme
of things.
> Are there any additional critical shortfalls? I know there is no
> shortage of ways we can help, but I want to start with the severe
> needs first.
That was all I had off the top of my head, as I said, I'm happy to
answer any other specific questions :)
You have been very helpful. I think we have enough to get started. Thanks again for all of
your insight!
jeff
--
Jeff Backus
jeff.backus(a)gmail.com
http://github.com/jsbackus
http://gitlab.com/jsbackus