On 26.02.2016 13:28, Daniel Mack wrote:
Hi Stef,
On 02/24/2016 04:52 PM, Stef Walter wrote:
> The first thing to note is that you're running the Cockpit unit tests.
> These barely interact with systemd.
>
> The integration tests are where the actual testing of cockpit features,
> such as the systemd interaction happens.
>
> Running the integration tests are documented here:
>
>
https://github.com/cockpit-project/cockpit/blob/master/test/README
>
> To try this out, install the stuff documented at the top of the file.
> And then you would do something like this. Run as non-root:
>
> $ cd test
> $ sudo ./vm-prep # only necessary the first time
> $ ./check-verify # runs the entire test suite
>
> But since you want a custom systemd, and not the entire test suite, you
> would probably end up moving towards something like this:
>
> $ TEST_OS=fedora-23
> $ ./vm-reset
> $ ./vm-download $TEST_OS
> $ ./vm-install --quick
> $ ./vm-customize -i systemd*.rpm # Or similar source of binaries
> $ ./verify/check-system-info # Just one of the tests
>
> You'll note that this is downloading an image (or in the former case) a
> few images. Jan Scotka has been running this these tests with custom
> images ... certainly possible. Update the link it the test/images directory.
Hmm, but the machines I'm running this one are more or less stateless,
and the scripts literally download gigabytes of images, which also takes
a long time.
Typically the runners that run these things cache the images, and only
new ones get downloaded. There's a $TEST_DATA environment variable that
htelps with this. On a test runner typically pointed to a volume that's
mounted persistently.
nd, only the first example I typed above, above downloads multiple
images. The second example just uses one.
But if you can't have a persistent volume, then that's a problem.
Luckily there's no need to use the ones we have available for
downloading. With a little scripting, such an image could be synthesized
by your test runner script. Take a look at test/guest/fedora-23.setup
for the kinds of things that are expected.
[1]
https://raw.githubusercontent.com/cockpit-project/cockpit/master/test/gue...
This would need more thinking if we would want to use that
as systemd test.
> Anyway ... you'll find that running these tests is a bit more complex
> and involved. But this is where the real continuous integration stuff
> happens.
Could you elaborate on why it has to be that complex?
Obviously if there's non-sensical complexity, we want to get that out of
the way. I'll do a bit of cleanup to help. But as to "why":
Cockpit is an interface for complex and messy operating systems. It
interacts directly with tons different projects and they all break,
regress, remove API and on and on.
So the integration tests have to actually integrate and boot real
operating systems. It ends up being very close to how users would use
Cockpit. In most cases they use the exact same build scripts (like spec
files) to build cockpit as ends up being built on that operating system
... and so on.
Without the integration testing that we do, the Cockpit project would be
dead ... or have a year long stabilization process before a release (ie:
as good as dead). But because the tests boot real operating systems,
that users are actually running, every week we can release something
that works, and catch regressions before they pile up, etc.
Obviously, I said above, you don't have to use the same operating system
images we do. If you want to generate your own test machine image,
that's possible.
I would have
thought about installing Cockpit onto the test machine and then run some
scripts that remote-control the local web interface via tools such as
Cucumber or RTest. Wouldn't something like that suffice and make the
test setup a whole lot easier to use?
phantomjs is what we use as the remote-control. And you could generate a
test machine that works with the given tests. I know that Jan Scotka
does. And it goes without saying, that if there's issues that get in the
way of rolling your own test machine, then we'd be happy to work with
you to change things.
Are you fine generating a qcow2 image? If so you would symlink/place it
into the test/images directory, and run a test like this:
$ TEST_OS=my-test-image ./verify/check-system-info
If you want to use a test machine you've started elsewhere, then there's
probably a couple things that need to be changed in test/testlib.py
(included by files like test/verify/check-system-info) to make that work
... and allow just specifying an address to test against, rather than
expecting to start a VM.
Let me know if you want to do this second option, it should be easy to
make that change.
Stef