On Tue, Feb 21, 2017 at 5:08 AM, Benjamin Berg <bberg(a)redhat.com> wrote:
Hi,
we are currently looking into enabling us to test laptops more
effectively. There are two main parts to the issue, which is to
1. have a system to run semi-automated tests on a standalone machine
and submit the results to an online server ("Fedora Tested Laptops")
and to
2. run parts of the tests in a fully automated fashion in a lab here in
Munich.
For now I am probably going to concentrate on the first part, but full
automation is still something to keep in mind. Some automation might
also happen without a full CI setup (e.g. simulate the lid switch or
plugging in different monitors using the chameleon board).
Focusing on the feature set the test runner should have, I see the
following requirements:
* Online submission of results
- Initially probably just manual updates and uploads to the wiki
- Fedora has resultsdb, but it is not designed to store larger blobs
* Ability to run standalone on a machine
- Resume test after interruptions like kernel panics
- Show tests status and user instructions for tests requiring
interaction, but allow the test to run automated when servo is
available.
- Allow skipping any tests requiring user interaction
* Possible to integrate into a CI setup
* Gathering of data about hardware before and during the test
- e.g. dmidecode, power usage, CPU states, firmware tests
So far I had a closer look at the at the following tools:
* OpenQA (
http://open.qa/)
* autotest (python,
http://autotest.github.io/)
* avocado (python,
https://avocado-framework.github.io/)
* resultsdb
* taskotron
Might I suggest that you also look into Restraint[1] before you make
your
final decision,
[1]
https://restraint.readthedocs.io/en/latest/
>
> Right now I think that avocado (a successor for autotest) is the best
> fit and can be adapted to the above needs. The only real advantage of
> autotest is that Google uses it on a large scale for testing
> chromebooks, but it seems harder to adapt and use. Most of the other
> tools cover other parts of a CI infrastructure.
>
>
> With this in mind, my current plan would be to work on the following
> items using avocado as a base:
>
> 1. Integrate a test status plugin including the ability to prompt for
> fine grained user instructions (maybe using DBus)
> 2. Work on support to resume interrupted runs (i.e. kernel panic)
> 3. Create data collection plugins and add features where sensible (e.g.
> maybe add RAPL power monitoring into upower)
> 4. Start writing test cases to exercise the above
>
> Opinions? Have I missed something important maybe?
>
> Benjamin
> _______________________________________________
> qa-devel mailing list -- qa-devel(a)lists.fedoraproject.org
> To unsubscribe send an email to qa-devel-leave(a)lists.fedoraproject.org