Hi,
we are currently looking into enabling us to test laptops more
effectively. There are two main parts to the issue, which is to
1. have a system to run semi-automated tests on a standalone machine
and submit the results to an online server ("Fedora Tested Laptops")
and to
2. run parts of the tests in a fully automated fashion in a lab here in
Munich.
For now I am probably going to concentrate on the first part, but full
automation is still something to keep in mind. Some automation might
also happen without a full CI setup (e.g. simulate the lid switch or
plugging in different monitors using the chameleon board).
Focusing on the feature set the test runner should have, I see the
following requirements:
* Online submission of results
- Initially probably just manual updates and uploads to the wiki
- Fedora has resultsdb, but it is not designed to store larger blobs
* Ability to run standalone on a machine
- Resume test after interruptions like kernel panics
- Show tests status and user instructions for tests requiring
interaction, but allow the test to run automated when servo is
available.
- Allow skipping any tests requiring user interaction
* Possible to integrate into a CI setup
* Gathering of data about hardware before and during the test
- e.g. dmidecode, power usage, CPU states, firmware tests
So far I had a closer look at the at the following tools:
* OpenQA (
http://open.qa/)
* autotest (python, http://autotest.github.io/)
* avocado (python, https://avocado-framework.github.io/)
* resultsdb
* taskotron
You might look into phoronix test suite. It's a hardware benchmarking framework, so it
should have decent capabilities to collect hw-related info, it's automatable, it can
upload results (but I don't know if the server part of this is open source, by default
the tool reports to
openbenchmarking.com, you need to investigate whether you can set up
your own server), it can track progress (in CLI and as a web page), it's packaged in
Fedora. I'm not sure it's an absolutely good fit, but probably worth looking
into.