CC'ing to cockpit-devel mailing list.
On 24.02.2016 13:57, Daniel Mack wrote:
In Brno, we've briefly talked about using Cockpit as extensive
systemd
DBus API tester. I wanted to follow up on that idea, and prepared a test
framework that runs on the CentOS CI infrastructure. This CI test is not
yet linked to our GitHub project, but I'm working on that.
The script provisions a host from a pool of bare metal machines with
CentOS 7, checks out systemd from a given PR branch, builds it, installs
it, and reboots the machine. Once it's back up, it SSHs back into it and
then checks out and builds Cockpit. The bits related to Cockpit are here:
https://github.com/systemd/systemd-centos-ci/blob/master/slave/cockpit.sh
As you can see, I'm using a fork of cockpit here, which has one extra
patch to disable 3 tests that failed in my test runs:
https://github.com/systemd/cockpit/commit/750f8702
Ideally, I would like to get rid of that and use the official upstream
with a specific tag of course. Without this extra patch, however, a
'make check' based on the 0.95 tag results in the following log:
http://zonque.org/cockpit-test-suite.log
Probably caused by running 'make check' as root. Can try and fix this,
or disable the test in that situation.
Kerberos is certainly outside of the scope of what we want to test,
so
it might be a good idea to make such tests fail gracefully if the server
can't be reached, or make them optional in some other way.
In total, the script currently takes around 4 minutes, 45 seconds to
complete. While that's not a whole lot, I wonder whether we can speed it
up in some way by only building or running a subset of the cockpit code.
After all, all we want to test is the systemd DBus API, everything else
can be omitted. Is there a sane way to limit the tests to just this?
The first thing to note is that you're running the Cockpit unit tests.
These barely interact with systemd.
The integration tests are where the actual testing of cockpit features,
such as the systemd interaction happens.
Running the integration tests are documented here:
https://github.com/cockpit-project/cockpit/blob/master/test/README
To try this out, install the stuff documented at the top of the file.
And then you would do something like this. Run as non-root:
$ cd test
$ sudo ./vm-prep # only necessary the first time
$ ./check-verify # runs the entire test suite
But since you want a custom systemd, and not the entire test suite, you
would probably end up moving towards something like this:
$ TEST_OS=fedora-23
$ ./vm-reset
$ ./vm-download $TEST_OS
$ ./vm-install --quick
$ ./vm-customize -i systemd*.rpm # Or similar source of binaries
$ ./verify/check-system-info # Just one of the tests
You'll note that this is downloading an image (or in the former case) a
few images. Jan Scotka has been running this these tests with custom
images ... certainly possible. Update the link it the test/images directory.
Anyway ... you'll find that running these tests is a bit more complex
and involved. But this is where the real continuous integration stuff
happens.
Other projects are starting to run some of them to test various things,
and if there's big issues making them work, we can try and solve the
issues. Poke someone in #cockpit if you run into trouble.
Stef